3:55 pm

Live Chat Unavailable Tuesday, December 16th, 2am to Wednesday, December 17th, 2am PST

Posted (December 15th, 2014 at 3:55 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Tuesday, December 16th, from 2am to 2am PST (the following morning, 24hours). We will be involving our support staff in our year’s end team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel this meeting will only help to improve our Live Chat team’s performance. This was previously announced as being between Tuesday December 16th at 12am till Wednesday, December 17th at 12am, but has been revised to provide the best coverage possible based on our staff’s schedules.

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 2am PDT the next morning, December 17th. Thank you for your patience and understanding.

This entry was posted on Monday, December 15th, 2014 at 3:55 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled DreamCompute Maintenance – December 4th 11am PST (4hrs) (RESOLVED)
2:59 pm

Scheduled DreamCompute Maintenance – December 4th 11am PST (4hrs) (RESOLVED)

Posted (December 2nd, 2014 at 2:59 pm PST) by andrea

On Thursday December 4th from 11AM to 3PM, our cloud engineering team will be investigating a known, intermittent connectivity issue. The team will be working in coordination with one of our networking vendors to diagnose the matter and plan a resolution. We expect network availability to remain consistent, though there is a possibility your instances will be temporarily unavailable during the maintenance window.

We will closely monitor the maintenance and will be posting periodic updates. This maintenance window only affects the DreamCompute service.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 4th, 2014, 12:45PM PDT: The maintenance has been completed. We performed some upgrades and first tests show the expected results. We will continue to test, evaluate and see if any further changes are needed in future maintenance periods. If you see any issues with your DreamCompute service after this maintenance please contact support.

Update December 4th, 2014, 12:30PM PDT: At this point we have upgraded software on 2 of our 6 network switches. Due to the redundant nature there appears to be no downtime related to this change. We will continue the upgrades and investigation. If you see issues that could be related to this maintenance please contact support.

Update December 4th, 2014, 12:00PM PDT: Maintenance investigation continues. No restarts or changes have been made yet that would cause any issues. We will continue to monitor the situation and update you through the maintenance window. – Justin

Update December 4th, 2014, 11:30AM PDT: Preparations for the investigation and upgrade are still ongoing at this time. We expect to start the actual changes within the next 30 minutes. – Justin

This entry was posted on Tuesday, December 2nd, 2014 at 2:59 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled DreamObjects Maintenance – December 9th 10am PST (24hrs) (RESOLVED)
2:12 pm

Scheduled DreamObjects Maintenance – December 9th 10am PST (24hrs) (RESOLVED)

Posted (December 2nd, 2014 at 2:12 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Tuesday December 9th from 10am to Wednesday December 10th 10am, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. Please rest assured that your data is safe and will not be affected.

The purpose of this maintenance is to update the configuration of Ceph, the underlying storage technology powering DreamObjects. The team is making the configuration change to prepare the cluster for additional capacity. The change will trigger the cluster to re-optimize the arrangement of its data and is likely to cause inconsistent availability. The bulk of the optimization will complete within the first few hours and will diminish throughout the day.

We will closely monitor the maintenance and will be posting periodic updates. This maintenance window only affects DreamObjects and the manual account backup feature.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 10th, 2014, 10:00AM PDT: The maintenance window is now closed, and the maintenance is complete. We did not detect any issues with the DreamObjects service during the last 24 hours and we are now one step closer to an upgraded and expanded DreamObjects cluster. If you notice any issues since the maintenance please contact support and we would be happy to assist you further. – Justin

Update December 9th, 2014, 15:00PM PDT: The maintenance window remains open and we are monitoring the cluster. There have been no issues to report after the configuration changes, and we are keeping the window open just in case. If you notice any issues please contact support. – Justin

Update December 9th, 2014, 12:00PM PDT: The maintenance window remains open and the cluster is still adjusting to the configuration changes. All services are reporting up and working normally and we will continue to monitor the situation. – Justin

Update December 9th, 2014, 10:50AM PDT: The maintenance continues. The cluster has adjusted to the configuration changes exactly as expected so far, and will continue to adjust to the changes the next few hours. – Justin

Update December 9th, 2014, 10:20AM PDT: The maintenance has begun. The configuration changes we wanted to make have been applied and we are monitoring the cluster as it adjusts to this change. We will update this post again in regular intervals or sooner if there is any information to pass along. – Justin

This entry was posted on Tuesday, December 2nd, 2014 at 2:12 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled DreamObjects Maintenance – December 2nd 10am PST (24hrs) (POSTPONED – see new post above)
3:29 pm

Scheduled DreamObjects Maintenance – December 2nd 10am PST (24hrs) (POSTPONED – see new post above)

Posted (November 25th, 2014 at 3:29 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Monday December 1st from 10am to Tuesday December 2nd 10am, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. Please rest assured that your data is safe and will not be affected.

The purpose of this maintenance is to update the configuration of Ceph, the underlying storage technology powering DreamObjects. The team is making the configuration change to prepare the cluster for additional capacity. A consequence of this change is an expected rebalance of the data to optimize the storage cluster. The bulk of the rebalance and optimization will complete within the first few hours and will diminish throughout the day.

We will closely monitor the maintenance and will be posting periodic updates.
Please note, this maintenance was originally scheduled for December 1st, 10am PST. However, based upon feedback, we delayed it to the 2nd (Tuesday) to minimize any potential impact for our customers.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 2nd, 2014, 2PM PST: Apologies for the late notice: this maintenance has been postponed until next week. There is additional prep work to complete before making the configuration change and it is not yet concluded. We’ll be adding a new status post with the details.

This entry was posted on Tuesday, November 25th, 2014 at 3:29 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 4.0.1 – Security Release
11:54 am

WordPress 4.0.1 – Security Release

Posted (November 20th, 2014 at 11:54 am PST) by Mika E.

WordPress 4.0.1 is now available as a security release for all previous versions of WordPress, and is being applied now to all sites who allow for Background Updates in WordPress, as well as any who use our Auto Update via Panel.

In fact… you may have already been emailed about this by your install.

You can read the release notes for 4.0.1, and this release is coordinated with a Drupal release, as both projects share a potential vulnerability (due to an issue in PHP) that could cause a denial of service attack. The upgrade applies to WordPress versions 4.0.1, 3.9.3, 3.8.5, and 3.7.5.

As a reminder, WordPress itself performs background updates of all minor releases, since WordPress 3.7 was released in 2013. That means even if you have disabled auto updates via panel (which you’d never do, because updating is the best thing since sliced bread), WordPress will still upgrade your 3.7, 3.8, and 3.9 installs to the latest, most secure, version.

If for whatever reason you went and disabled all possible update methods to your site, please log in to your WordPress site now, head to Dashboard → Updates and simply click “Update Now.” It’s just that easy.

Speaking of updates …. WordPress 4.1 is due out in December! Now’s the time to have a look at your sites and make sure all your plugins and themes are up-to-date in preparation for some cool new features. You may even want to upgrade your PHP settings in panel to PHP 5.4, just to be extra cool.

This entry was posted on Thursday, November 20th, 2014 at 11:54 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency maintenance on shared web server ‘fullback’ [RESOLVED]
6:36 am

Emergency maintenance on shared web server ‘fullback’ [RESOLVED]

Posted (November 17th, 2014 at 6:36 am PST) by chalise

Our shared web server, ‘fullback’ is in need of an emergency memory replacement, which will be starting shortly.

We expect the maintenance window to last about a half hour and we will be updating this status post with updates as needed.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, or regarding any other services, please submit a support ticket through DreamHost panel with your details so we can help!

Update – 6:57 am PST: Maintenance is complete! All services on ‘fullback’ are back up and running. Thank you for your patience!

This entry was posted on Monday, November 17th, 2014 at 6:36 am and is filed under Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled DreamCompute Maintenance – November 17th, 6AM PST (30 min) (RESOLVED)
11:16 am

Scheduled DreamCompute Maintenance – November 17th, 6AM PST (30 min) (RESOLVED)

Posted (November 14th, 2014 at 11:16 am PST) by andrea

As part of our continued efforts to improve our cloud services, on Monday, November 17th, from 6am-6:30am PST, our cloud engineering team will be performing maintenance on our DreamCompute service. The maintenance work is minor and is limited to a configuration change for the compute and network component services. During this maintenance window, you may experience short periods of connectivity loss to your instances. We apologize for any inconvenience this may cause.

Update November 17th, 2014, 7:44AM PST: The changes for this maintenance are complete and our testing shows the system working normally. This maintenance is now complete. Should you still run into any issues with DreamCompute, please report it to tech support and we will get it checked out immediately. – Justin

Update November 17th, 2014, 6:30AM PST: Work is continuing on this configuration change, but we do not expect any further disruption in service for customers. We will keep this status post open until all work is complete so that customers are aware and can report issues to tech support if any do arise. Thanks for your patience during this maintenance process. – Justin

This entry was posted on Friday, November 14th, 2014 at 11:16 am and is filed under System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Network Maintenance for Homiemail-sub5 Email Cluster and subset of Mysql Servers in US-West (Los Angeles) Data Center – November 6th, 10pm PST (4hrs) (RESOLVED)
11:41 am

Scheduled Network Maintenance for Homiemail-sub5 Email Cluster and subset of Mysql Servers in US-West (Los Angeles) Data Center – November 6th, 10pm PST (4hrs) (RESOLVED)

Posted (November 5th, 2014 at 11:41 am PST) by andrea

We will be conducting maintenance in our US-West (Los Angeles) Data Center starting at 10pm PST on November 6th. The purpose of the maintenance is to upgrade the capacity of the networking hardware to these machines and improve overall redundancy. We are scheduling a window of four hours for thorough pre and post testing, but the actual service interruption will be minimal. We will be closely monitoring the process and will be posting updates as we progress.

Please note, this maintenance affects a subset of mysql servers (see below for the full list) and the mail traffic on homiemail-sub5. No mail will be lost only queued up for later delivery. You can tell if you’re affected by checking for your mysql server and email cluster under the Account Status in your DreamHost panel. We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

The list of affected mysql servers:
straptail
stonefish
stonecat
surgeonfish
stingfish
stickleback
switch
switch
sturgeon
sunfish
surfperch
steelhead
stargazer
squirrelfish
squaretail
squawfish
springfish
squeaker

Update November 7th, 2014, 12:45AM PST: We’re happy to report that maintenance is now complete. If you need any assistance related to this or anything else, please let us know by contacting tech support.

Update November 6th, 2014, 11:38PM PST: We are continuing with the maintenance as scheduled, and all the steps are being completed without impact on services. We appreciate you continued patience, and we’ll post another update within the hour or sooner if there is a status change.

Update November 6th, 2014, 10:38PM PST: So far so good! We will update this post again within the hour or if there is a status change.

Update November 6th, 2014, 10:05PM PST: We have started the maintenance as scheduled. Please keep an eye out for updates as we progress through the steps.

This entry was posted on Wednesday, November 5th, 2014 at 11:41 am and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 187 shared web hosting servers to Ubuntu 12.04 Precise – November 8th, 8PM – 2AM PST(RESOLVED)
2:01 pm

Upgrading 187 shared web hosting servers to Ubuntu 12.04 Precise – November 8th, 8PM – 2AM PST(RESOLVED)

Posted (November 4th, 2014 at 2:01 pm PST) by Brandon

We’re completing our roll-out of Ubuntu 12.04 Precise to the final 187 shared web servers  this Saturday, November 8th.  While the total estimated maintenance is 6 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order. While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.   Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

abhainndearg
abraham-clark
afzelia
agave
agrajag
allitnil
andy
appling
ardbeg
aten
atum
auchentoshan
balin
balmenach
baratheon
bard
benrinnes
beorn
bernstein
bifur
blackboard
bocote
bofur
bowmore
brunswick
buckshot
bulldog
bungalow
butts
cactus
candler
carter-braxton
catoosa
cavalier
cement
charles-cotesworth
chausson
chavez
checkerboard
checkmate
chesterfield
chrysanthemum
clam
clarke
daily
dancer
daylily
decatur
deckhand
delpuerto
dickenson
dindy
dinwiddie
dodge
dori
dwalin
early
echols
edradour
effingham
evans
fauquier
finley
fireplug
fistfight
foxcraft
franklin
george-ross
george-wythe
glen-elgin
glen-grant
glendronach
glenfiddich
glengoyne
glengyle
glenkinchie
glenlivet
glenlossie
glentauchers
gloucester
goochland
grady
halifax
hancock
hanson
hapi
hart
heard
heimdallr
hodhr
hoenir
horsetail
hovhaness
hugh-williamson
invergordon
isle-of-wight
kininvie
kittyhawk
kneecap
knockdhu
lanier
lumpkin
lyman-hall
maat
macon
madison
mahler
matthew-thornton
meriwether
mignone
miltonduff
mirfak
monroe
montgomery
moonflower
naos
nelson
nihal
nottoway
oin
page
paulding
phact
pierce
poplar
prince-george
prince-william
ptah
pulteney
quitman
ra
rana
reshep
richard-dobbs
richmond
rimsky-korsakov
roanoke
rockbridge
roseisle
russell
sadira
saint-saens
santoro
sawhorse
scapa
scheat
screven
segin
shenandoah
sheoak
signature
skymaster
smelter
snotra
spectator
spotsylvania
springbank
stanford
starlaw
sterope-ii
strathisla
strathmill
stutter
sugarfoot
tamdhu
tazewell
telfair
the-macallan
thomas-heyward-jr
tinkerbell
tool
tracer
transit
trapline
treasure
twiggs
union
vroomfondel
walker
warren
westmoreland
williamson
wise
wonko
york
zem
zwilich
If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have! While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available.
While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.
Update November 9th, 2014, 12:50AM PST: All shared web servers have now been upgraded to Ubuntu 12.04 Precise. Thanks for your patience throughout this process!
Update November 9th, 2014, 12:00AM PST: All servers from the first batch (“abhainndearg” to “checkmate”) and Batch #2 (“chesterfield” through “gloucester”) have completed the upgrade to Ubuntu. All servers from batch #3 (“goochland” through “oin”) are now online except for “lanier”, “matthew-thornton”. All servers from  batch #4 (“page” through “sterope-ii”) are complete. Batch #5 (“strathisla” through “zwilich”) web servers are beginning to come online now.
Update November 8th, 2014, 11:51PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) and Batch #2 (“chesterfield” through “gloucester”) have completed the upgrade to Ubuntu. All servers from batch #3 (“goochland” through “oin”) are now online except for “lanier”, “matthew-thornton”. All servers from  batch #4 (“page” through “sterope-ii”) are online except for “page”. Batch #5 (“strathisla” through “zwilich”) web servers are beginning to come online now.
Update November 8th, 2014, 11:28PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) and Batch #2 (“chesterfield” through “gloucester”) have completed the upgrade to Ubuntu. All servers from batch #3 (“goochland” through “oin”) are now online except for “lanier”, “matthew-thornton”, “nottoway”. All servers from  batch #4 (“page” through “sterope-ii”) are online except for “page”, “pierce”, “prince-george”. Batch #5 (“strathisla” through “zwilich”) is is continuing OS installation.
Update November 8th, 2014, 11:05PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) and Batch #2 (“chesterfield” through “gloucester”) have completed the upgrade to Ubuntu. Sites on batch #3 (“goochland” through “oin”) are now beginning to come online. Batch #4 (“page” through “sterope-ii”) is now beginning restoration &  configuration. Batch #5 (“strathisla” through “zwilich”) is now beginning the OS installation phase.
Update November 8th, 2014, 10:51PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) and Batch #2 (“chesterfield” through “gloucester”) have completed the upgrade to Ubuntu. Sites on batch #3 (“goochland” through “oin”) is now beginning the restoration phase. Batch #4 (“page” through “sterope-ii”) is now beginning OS installation.
Update November 8th, 2014, 10:25PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed the upgrade to Ubuntu. Batch #2 (“chesterfield” through “gloucester”) is complete other than the following servers: “daily”, “deckhand”, “dodge”, “echols”, “evans”, “fireplug”, “fistfight”, “foxcraft”. Batch #3 (“goochland” through “oin”) is now continuing configurations. Batch #4 (“page” through “sterope-ii”) is now beginning OS installation.
Update November 8th, 2014, 10:17PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed the upgrade to Ubuntu. Batch #2 (“chesterfield” through “gloucester”) is complete other than the following servers: “daily”, “deckhand”, “dodge”, “echols”, “evans”, “fireplug”, “fistfight”, “foxcraft”. Batch #3 (“goochland” through “oin”) is now beginning the OS installation phase.
Update November 8th, 2014, 10:12PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed the upgrade to Ubuntu. Sites from Batch #2 (“chesterfield” through “gloucester”) are now beginning to come back online. Batch #3 (“goochland” through “oin”) is now beginning the OS installation phase.
Update November 8th, 2014, 10:05PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed other than “agrajag” . Sites from Batch #2 (“chesterfield” through “gloucester”) are now beginning to come back online. Batch #3 (“goochland” through “oin”) is now beginning the OS installation phase.
Update November 8th, 2014, 9:46PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed other than “agrajag” . Sites from Batch #2 (“chesterfield” through “gloucester”) are now beginning to come back online.
Update November 8th, 2014, 9:40PM PST: All servers from the first batch (“abhainndearg” to “checkmate”) have completed other than the following servers: “andy”, “agrajag”, “baratheon”, and “blackboard”. Batch #2 (“chesterfield” through “gloucester” is continuing OS installation.
Update November 8th, 2014, 9:21PM PST: Sites on the first batch (“abhainndearg to “checkmate”) are now beginning to come back online under Ubuntu. The second batch (“chesterfield” through “gloucester”) is now beginning the OS installation phase.
Update November 8th, 2014, 8:56PM PST: Sites on the first batch (“abhainndearg to “checkmate”) are now beginning to come back online under Ubuntu.
Update November 8th, 2014, 8:39PM PST: The first batch (“abhainndearg to “checkmate”) is now entering the configuration phase.
Update November 8th, 2014, 8:11PM PST: We’re running 5 batches of Ubuntu upgrades tonight. We’ve finished the backup phase for all  the batches, and are now entering the OS installation phase  for the first batch (“abhainndearg to “checkmate”).
This entry was posted on Tuesday, November 4th, 2014 at 2:01 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Upgrading 187 shared web hosting servers to Ubuntu 12.04 Precise – November 8th, 8PM – 2AM PST(RESOLVED)”

“We’re running 5 batches of Ubuntu upgrades tonight. We’ve finished the backup phase for all the batches”

So if my server in the last batch, any file created/changed starting now and till the site is taken offline will be lost? I’m concerned especially about static files which cache database queries, which will be inconsistent with the real data.

@David These are precautionary backups and configuration backups. Your data under /home will be untouched. :)

server won’t parse php in .html files or .htm files anymore. What do I add in htaccess to enable this?

@sheriden: This isn’t a known issue, please submit a support ticket and our staff will be in touch ASAP.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Saturday October 25th 4pm – Sunday October 26th 4pm PDT
7:57 pm

Live Chat Unavailable Saturday October 25th 4pm – Sunday October 26th 4pm PDT

Posted (October 23rd, 2014 at 7:57 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Saturday, October 25th, from 4pm to 4pm PDT (the following afternoon). We will be involving our support staff in our annual Halloween party, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 4pm PDT the next afternoon, Oct 26th. We thank you for your patience and understanding.

This entry was posted on Thursday, October 23rd, 2014 at 7:57 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Live Chat Unavailable Saturday October 25th 4pm – Sunday October 26th 4pm PDT”

This is extremely refreshing and great to see. A company who cares and rewards their hardworking staff! Kudos!

did they take wiki.dreamhost.com with them?

Dear Dreamhost.

I am happy for your staff, unfortunately our site is down again (this happens now every month) for no explanation (we think it coincides with the billing cycle). The response from you on Twitter on Saturday was “It shows the account has been disabled for non payment. You can remedy that here https://panel.dreamhost.com/index.cgi?tree=billing.payment& … Thanks! *MC”

We signed in and received this notice:
Current Account Balance
Current Balance: $0.00
All paid up. Thank you!

As we have written before when this happened, this reflects badly on our site (300,000 unique visitors per month) and on you as a server.

Can you fix this please.

Sean

Customer support = epic fail

Cool Brian,
Fingers crossed you post some pictures from the Halloween Party!
Robdog

Thanks a lot for the blog.Really thank you! Keep writing.

I really enjoy the article.Really looking forward to read more. Cool.

I loved your blog article.Really looking forward to read more. Keep writing.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » HTTPS Web Site Owners – SSL v.3 Disablement
5:36 pm

HTTPS Web Site Owners – SSL v.3 Disablement

Posted (October 16th, 2014 at 5:36 pm PST) by Brian@dreamhost

DreamHost has disabled SSL v.3 across our network. After 18 years of being available as an option for encryption & security in browser communications, the SSL 3.0 protocol has finally met it’s demise in the form of a not so cuddly POODLE (http://en.wikipedia.org/wiki/POODLE) attack.

DreamHosters with HTTPS enabled on their web sites are the only customers that will be affected. You can verify if you have SSL enabled on your domain, here. Visitors whose web browsers do not support the more modern TLS protocol will not be able to negotiate a Secure connection.

The good news is those antique browsers are largely retired from use. The most modern browser that will be unable to negotiate a secure connection with SSL v.3’s disablement is Internet Explorer 6.0 with Windows NT 4.0, released Aug 2001. Please don’t hesitate to contact our support team with any further questions in this matter.

This entry was posted on Thursday, October 16th, 2014 at 5:36 pm and is filed under Announcement, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Service Connectivity Issues (RESOLVED)
8:29 am

DreamObjects Service Connectivity Issues (RESOLVED)

Posted (October 16th, 2014 at 8:29 am PST) by Dan J

Our cloud engineering team is investigating connectivity issues being reported by the DreamObjects service. We do not currently have an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe and we will continue to update this post as we gather more information.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update October 16th, 2014, 9:50AM PDT:DreamObjects service continues to be working and work continues on backend hardware replacement and improvements. We will continue to monitor our systems for further issues.

Update October 16th, 2014, 9:20AM PDT:Service has been restored at this time, and work is continuing on the backend to fix remaining issues and replace some hardware. We will continue to monitor our systems for further issues.

Update October 16th, 2014, 9:00AM PDT: Our team of cloud administrators and developers are continuing to work on the issue and are making progress. We apologize for the issues with DreamObjects this morning and appreciate your patience while we resolve this issue.

This entry was posted on Thursday, October 16th, 2014 at 8:29 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 18th, 8PM – 1AM PDT (RESOLVED)
12:29 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 18th, 8PM – 1AM PDT (RESOLVED)

Posted (October 14th, 2014 at 12:29 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 18th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

aberfeldy
aberlour
aker
alltabhainne
ammit
andriessen
apophis
ardmore
arran
atterberg
atum
auchinblae
auchroisk
bakha
balbair
bat
beech
bennevis
benriach
benromach
bizet
blacktop
bladnoch
blairathol
bliss
brackla
braeval
brian
britten
broadside
bruckner
bruichladdich
buckingham
cameronbridge
carroll
chamomile
charles-city
chatham
cherry
cloudburst
corydalis
cragganmore
craigellachie
dade
dagr
dalmore
dalwhinnie
danielpour
davies
deanston
debussy
deepthought
delling
draeseke
eddie
effingham
fan
fettercairn
fireplug
forseti
franck
geb
gillis
girvan
glascock
glazunov
glen-garioch
glen-moray
glen-scotia
glen-spey
glenallachie
glencadam
glendullan
glenfarclas
glenglassaugh
glenmorangie
glenrothes
glenturret
gloin
gmot
goldmark
guerra-peixe
haralson
hartmann
hathor
heget
hickory
highland-park
hindemith
holmboe
iounn
isle-of-jura
joro
kellaa
khnum
khonsu
kilchoman
king-george
knockando
lalo
linkwood
lochnagar
longmorn
macduff
madison
mafdet
magni
mannochmore
marblewood
melartin
messiaen
mimir
mortlach
mozart
muliphein
nielsen
nighthawk
nystroem
oban
odin
pallando
penderecki
persichetti
porrima
radagast
ras-alhague
rubbra
saga
sarir
saruman
sasin
satis
schubert
shaula
sif
sjofn
springtime
stagecoach
still
strathclyde
strauss
stravinsky
suhail
suk
sumter
tienkwan
tobermory
toch
tournemire
tubin
tullibardine
vivaldi
wagner
wheels
widor
wolfburn

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 19th, 2014, 12:18AM PDT: The Ubuntu upgrades have completed and all batches are now back online. The following web servers were not upgraded, and will be upgraded at later date: zem vroomfondel baratheon effingham fireplug.

Update October 18th, 2014, 11:41PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete. Batch #4(“persichetti” through “wolfburn”) is complete other than”tienkwan”.

Update October 18th, 2014, 11:31PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete. Batch #4(“persichetti” through “wolfburn”) is complete other than “stagecoach”, “wheels”, “tienkwan”, “saruman”, “strauss”.

Update October 18th, 2014, 11:22PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete other than “pallando”. Batch #4(“persichetti” through “wolfburn”) is complete other than “stagecoach”, “wheels”, “tienkwan”, “saruman”, “strauss”.

Update October 18th, 2014, 11:05PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete other than “pallando”. Batch #4(“persichetti” through “wolfburn”) has completed the OS installation phase, and is entering the final configuration stage.

Update October 18th, 2014, 10:48PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete other than haralson, madfet, pallando, king-george.. Batch #4(“persichetti” through “wolfburn”) has finished the backup phase and has entered the OS installation phase.

Update October 18th, 2014, 10:35PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is complete other than”davies”. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete other than haralson, madfet, pallando, king-george.. Batch #4(“persichetti” through “wolfburn”) has finished the backup phase and has entered the OS installation phase.

Update October 18th, 2014, 10:26PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is complete other than “glascock” and “davies”. Batch #3(“guerra-peixe” through “penderecki”) machines are beginning to come back online. Batch #4(“persichetti” through “wolfburn”) has finished the backup phase and has entered the OS installation phase.

Update October 18th, 2014, 10:13PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is complete other than “glascock” and “davies”. Batch #3(“guerra-peixe” through “penderecki”) machines are beginning to come back online. Batch #4(“persichetti” through “wolfburn”) has begun the backup phase.

Update October 18th, 2014, 10:04PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is complete other than “glascock” and “davies”. Batch #3(“guerra-peixe” through “penderecki”) machines are beginning to come back online.

Update October 18th, 2014, 9:54PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is complete other than “glascock” and “davies”. Batch #3(“guerra-peixe” through “penderecki”) is nearing completion of the OS installation phase and will be entering the configuration phase.

Update October 18th, 2014, 9:45PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”)  is nearing completion and sites are coming online now. Batch #3 has begun the OS installation phase.

Update October 18th, 2014, 9:41PM PDT:Batch #3(“guerra-peixe” through “penderecki”) has now begun the OS installation phase.

Update October 18th, 2014, 9:38PM PDT: All servers from Batch #1 (“aberfeldy” through “cherry”) are now back online other than the following servers:

 

charles-city
carroll

Sites on Batch #2(“cloudburst” through “goldmark”) are now coming back online.

Update October 18th, 2014, 9:25PM PDT:Sites on Batch #1  (“aberfeldy” through “cherry”)  are now beginning to come back online post-upgrade. Batch #2 (“cloudburst” through “goldmark”) has finished the OS installation phase and is beginning final configurations.

Update October 18th, 2014, 9:07PM PDT:Sites on Batch #1  (“aberfeldy” through “cherry”)  are now all complete other than the 4 following servers:

aker
bliss
charles-city
carroll

 

Batch #2 (“cloudburst” through “goldmark”) has begun the OS installation phase.

Update October 18th, 2014, 9:02PM PDT: Sites on Batch #1  (“aberfeldy” through “cherry”)  are now beginning to come back online post-upgrade. Batch #2 (“cloudburst” through “goldmark”) has begun the backup phase.

Update October 18th, 2014, 8:55PM PDT: Batch #2 (“cloudburst” through “goldmark”) has begun the backup phase.

Update October 18th, 2014, 8:44PM PDT:  Batch #1 (“aberfeldy” through “cherry”) has finished the OS installation phase and is entering the configuration phase.

Update October 18th, 2014, 8:20PM PDT:  Batch #1 (“aberfeldy” through “cherry”) has finished the backup phase, and is now entering the OS installation stage.

Update October 18th, 2014, 8:13PM PDT: We’ll be running 4 batches of 40 web servers tonight. There are 3 major phases, which are the backup phase, the OS installation phase, and the configuration phase.  We’ve now begun the backup phase for Batch #1 (“aberfeldy” through “cherry”).

This entry was posted on Tuesday, October 14th, 2014 at 12:29 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 12th, 8PM – 1AM PDT (RESOLVED)
8:14 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 12th, 8PM – 1AM PDT (RESOLVED)

Posted (October 7th, 2014 at 8:14 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Sunday, October 12th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

abraham-baldwin
acrobat
albemarle
amelia
amherst
appomattox
arthur-dent
augusta
backseat
bamboo
bandbox
banks
bartow
baseball
bath
benjamin-harrison
bigtop
blacktop
blueprint
bombur
bookstore
bryan
buchanan
buckingham
bulloch
centurion
challenger
charles-pickney
charlotte
chatham
cherokee
claw
coachwood
coffee
columbia
cook
coweta
crawford
crisp
crystal
dain
daniel-carroll
deacon
diamond
dire
durin
eddie
effingham
elrond
essex
evergreen
fadeaway
fan
fannin
fayette
fencing
fireplug
flivver
ford-prefect
frederick
giles
glascock
gloin
glynn
gwinnett
habersham
hall
halo
hanover
haralson
henrico
henry
highland
hotshot
instructor
james-city
jenkins
josiah-bartlett
kili
king-william
lancaster
lee
long
loudoun
lunenburg
magnolia
marion
miller
morgan
morinehtar
nighthawk
nori
northumberland
patroller
pickens
pierce-butler
pike
pivot
polk
powhatan
prince-edward
putnam
randolph
regor
rockingham
sadalmelik
saiph
samuel-chase
sandbox
saruman
schley
scott
seminole
sheliak
skat
slacrl
snapshot
snowstorm
softpack
stardust
stewart
sugi
sussex
sycamore
talbot
templer
terrell
thrain
thror
tift
toombs
towns
tracker
traffic
trail
treutlen
tricia-mcmillan
tureis
walton
ware
warehouse
wheels
wilcox
wilkes
william-floyd
worth
wythe
zaphod-beeblebrox

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 13th, 2014, 12:19AM PDT: The maintenance has now completed. If you find that you’re experiencing any issues with your website(s) please contact support via the link above.

Update October 12th, 2014, 11:54PM PDT: ‘sugi’ is now back online, and only ‘tureis’ and ‘william-floyd’ remain.

Update October 12th, 2014, 11:50PM PDT: The web server ‘skat’ has completed it’s upgrade to Ubuntu and only ‘sugi’, ‘tureis’ and ‘william-floyd’ remain incomplete.

Update October 12th, 2014, 11:34PM PDT: All web servers have completed the upgrade to Ubuntu other than the following four:

 

skat
sugi
tureis
william-floyd

We expect them to be fully back on-line soon and will update this post when we have new information.

Update October 12th, 2014, 10:50PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) has completed and sites are back online. Sites on Batch #2 (‘crisp’ through ‘kili’) are all back online.

Batch #3 (‘king-william’ through ‘slacrl’) has completed other than the web server ‘skat’.

Servers from Batch #4 (‘snapshot’ through ‘zaphod-beeblebrox’) are now coming back online.

Update October 12th, 2014, 10:40PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) has completed and sites are back online. Sites on Batch #2 (‘crisp’ through ‘kili’) are all back online other than the following servers:

crystal

Batch #3 (‘king-william’ through ‘slacrl’) has completed other than the following servers:

nori
pivot
skat

Batch #4 (‘snapshot’ through ‘zaphod-beeblebrox’) is still completing OS installation.

Update October 12th, 2014, 10:05PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) has completed and sites are back online. Sites on Batch #2 (‘crisp’ through ‘kili’) are all back online other than the following servers:

crystal
fadeaway
ford-prefect
jenkins

Batch #3 (‘king-william’ through ‘slacrl’) has completed the backup phase and is beginning final configurations. Batch #4 (‘snapshot’ through ‘zaphod-beeblebrox’) has completed the backup phase and is beginning OS installation.

Update October 12th, 2014, 9:52PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) has completed and sites are back online. Sites on Batch #2 (‘crisp’ through ‘kili’) are beginning to come back online. Batch #3 (‘king-william’ through ‘slacrl’) has completed the backup phase and is beginning final configurations. Batch #4 (‘snapshot’ through ‘zaphod-beeblebrox’) is beginning the backup phase.

Update October 12th, 2014, 9:48PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) is complete other than the ‘charlotte’ web server, which is still running final configurations. Sites on Batch #2 (‘crisp’ through ‘kili’) are beginning to come back online. Batch #3 (‘king-william’ through ‘slacrl’) has completed the backup phase and is beginning final configurations. Batch #4 (‘snapshot’ through ‘zaphod-beeblebrox’) is beginning the backup phase.

Update October 12th, 2014, 9:34PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) is complete other than the ‘charlotte’ web server, which is still running final configurations. Sites on Batch #2 (‘crisp’ through ‘kili’) are beginning to come back online. Batch #3 (‘king-william’ through ‘slacrl’) has completed the backup phase and is now beginning the OS installation phase.

Update October 12th, 2014, 9:28PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) is complete other than the ‘charlotte’ web server, which is still running final configurations. Sites on Batch #2 (‘crisp’ through ‘kili’) are beginning to come back online. Batch #3 (‘king-william’ through ‘slacrl’) is now beginning the backup phase.

Update October 12th, 2014, 9:04PM PDT:Batch #1 (‘abraham-baldwin’ through ‘crawford’)  is now beginning to complete the configuration phase, and sites should be coming back online shortly. Batch #2 (‘crisp’ through ‘kili’) has completed the backup phase and is beginning the OS installation phase.

Update October 12th, 2014, 8:49PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) is now beginning to complete the OS installation phase, and the configuration stage is beginning. Batch #2 (‘crisp’ through ‘kili’) is now beginning the backup phase.

Update October 12th, 2014, 8:20PM PDT: Batch #1 (‘abraham-baldwin’ through ‘crawford’) has completed the backup phase and is now rebooting for OS installation.

Update October 12th, 2014, 8:12PM PDT: We’ll be running the upgrades in batches of 35 tonight. There will be 4 total batches for a total of 136 machines to  be upgraded. Batch #1 (‘abraham-baldwin’ through ‘crawford’) has just entered the backup phase. The following servers will not be upgraded due to PHP5.2 users still existing on them:

nighthawk
effingham
saruman
buckingham
fireplug
blacktop
fan
wheels
haralson
eddie
gloin

This entry was posted on Tuesday, October 7th, 2014 at 8:14 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 11th, 8PM – 1AM PDT (RESOLVED)
8:06 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 11th, 8PM – 1AM PDT (RESOLVED)

Posted (October 7th, 2014 at 8:06 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 11th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

accomack
alleghany
appling
arlington
atkinson
barrow
beehive
bellhop
ben-hill
berrien
bibb
birdseye
bland
bleckley
boardwalk
botetourt
brimstone
broadside
byrch
cableboy
calico
camden
cameo
campbell
candlestick
carbine
carnation
caroline
carpet
cartwheel
chandelier
chariot
charlton
chattooga
chessman
christopher
clay
clayton
coach
coppertone
culpeper
cumberland
dade
dasher
david-brearley
deepthought
derby
dog
dooly
dougherty
douglas
dusty
elbert
emanuel
fili
flag
flotus
fluvanna
footprint
forefinger
forsyth
frankie
fullback
fulton
gandalf
gargravarr
george-taylor
george-washington
gordon
grayson
greensville
gunning-bedford
horsehide
horsepower
houston
jacob-broom
james-mchenry
james-wilson
jeff-davis
jefferson
john-adams
john-hart
john-rutledge
king-and-queen
king-george
lock
lotus
louisa
lowndes
marine
mathews
mcduffie
mcintosh
mecklenburg
middlesex
mitchell
murray
muscogee
new-kent
newton
nicholas-gilman
northampton
oconee
oliver-wolcott
ori
peach
peninsula
philip-livingston
pinafore
pittsylvania
potus
professor
pulaski
rabun
radagast
radiant
rawhide
roadhouse
robert-morris
rosebush
samuel-adams
smyth
southampton
spalding
springtime
stafford
stagecoach
staircase
starlight
stephen-hopkins
stephens
store
sunburn
sunshine
surry
taliaferro
tattnall
taylor
thomas
thomas-fitzsimmons
thomas-nelson-jr
turner
volcano
washington
wayne
white
whitfield
wilkinson
william-paca
william-paterson
william-whipple

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 12th, 2014: 1:24AM PDT: The Ubuntu upgrade on the web server ‘chessman’ has completed. The maintenance has now completed. If you find that you’re experiencing any issues with your website(s) please contact support via the link above. The following web servers were not upgraded tonight, and will be upgraded at a future date:

 

stagecoach
king-george
radagast
deepthought
broadside
dade
springtime
culpeper

 

Update October 12th, 2014: 1:05AM PDT: All web servers from all batches other than ‘chessman’ have come back online. We expect this server to be on soon, and the maintenance will then be complete.

Update October 12th, 2014: 12:49AM PDT: All web servers from Batch #1 and Batch #2 are online. There are 5 web servers left from the remaining batches and we expect this maintenance to be completed shortly.

Update October 11th, 2014: 11:50PM PDT: All web servers from Batch #1 and Batch #2 are back up and running.

Update October 11th, 2014: 11:43PM PDT: All but one web server from Batch #1 is now online. 1 web server remains for Batch #2 and Batch #3 and Batch #4 are waiting on 7 each.

Update October 11th, 2014: 11:11PM PDT: All but one web server from Batch #1 is now online. 5 servers remain from Batch #2. Batch #3 is finishing up and Batch #4 is continuing with the OS installation phase.

Update October 11th, 2014: 10:45PM PDT: Almost all servers from Batch #1 and Batch #2 are online. Servers from Batch #3 are coming online now and Batch #4 has entered the OS installation phase.

Update October 11th, 2014: 10:32PM PDT: Almost all servers from Batch #1 and Batch#2 are online. Batch #3 is continuing the OS installation phase. Batch #4 (the final batch) has begun the backup phase.

Update October 11th, 2014: 10:25PM PDT: All but 10 servers from Batch #1 have completed installation. Servers on Batch #2 are beginning to come online now. Batch #3 has entered the OS installation phase.

Update October 11th, 2014: 9:53PM PDT: Websites from Batch #1 are continuing to come back online. Batch #2 is continuing OS installation and Batch #3 has started the backup phase.

Update October 11th, 2014: 9:44PM PDT: Websites from Batch #1 are now beginning to come back online. Batch #2 is progressing through OS installation.

Update October 11th, 2014: 9:30PM PDT: Batch #1 has completed OS installation and is now installing packages. Batch #2 will be starting OS installation momentarily.

Update October 11th, 2014: 9:11PM PDT:  The webserver ‘appling’ has been removed from tonight’s upgrades due to an active PHP5.2 user. Batch #2 has begun the backup phase, Batch #1 is still in the OS installation phase.

Update October 11th, 2014: 9:03PM PDT: The backup stage for batch #1 has completed, and the machines are rebooting into Ubuntu for OS installation.

 

Update October 11th, 2014: 8:29PM PDT: We’ll be running the upgrades in batches of 40, 40, 40 and 25 tonight. We’ve started the first phase for batch #1, the backup phase. The following web servers will not be upgraded tonight, as they still have active moves of PHP5.2 users:

stagecoach
king-george
radagast
deepthought
broadside
dade

springtime

culpeper

 

This entry was posted on Tuesday, October 7th, 2014 at 8:06 pm and is filed under Multiple Server Issue, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 5th, 8PM – 1AM PDT (RESOLVED)
11:31 am

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 5th, 8PM – 1AM PDT (RESOLVED)

Posted (September 30th, 2014 at 11:31 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Sunday, October 5th. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

alexander-hamilton
arthur-middleton
baker
baldwin
baratheon
bedford
benjamin-franklin
benjamin-rush
benjy
bilbo
blout
brooks
buckwell
button-gwinnett
bywater
caesar-rodney
calhoun
caliber
caravan
cargo
carousel
charles-carroll
chattahoochee
chelsted
clinch
cloverleaf
cobweb
colquitt
companion
cowpuncher
craig
crossbow
crown
curbside
daniel-jenifer
dawson
dekalb
driftwood
edward-rutledge
elbridge-gerry
falcon
fenchurch
fireside
flying
fog
forward
francis-hopkinson
francis-lewis
francis-lightfoot-lee
gag-halfrunt
george-clymer
george-read
george-walton
gilmer
gouverneur-morris
hactar
halfback
handshake
harris
headlight
hedgehog
holly
huntsman
irwin
james-madison-jr
james-smith
jared-ingersoll
jasper
jeltz
john-blair
john-dickinson
john-hancock
john-langdon
john-morton
john-penn
john-witherspoon
jonathan-dayton
joseph-hewes
lamar
laurens
lewis-morris
lightfoot
lintilla
lizard
majikthise
maximillian
oglethorpe
oolon-colluphid
pacemaker
pass
pavillion
pincushion
prak
principal
rainbow
rappahannock
redfern
reliant
rhyme
ribbon
richard-bassett
richard-henry-lee
richard-stockton
ridgeline
ringside
robert-treat-paine
rockdale
roger-sherman
roosta
rufus-king
samuel-huntington
sandstone
saturn
scorecard
searchlight
shared-with-ssds
sheepskin
shotgun
slartibartfast
snowbank
strawberry
sundance
supervisor
thomas-jefferson
thomas-lynch-jr
thomas-mckean
thomas-mifflin
thomas-stone
tiller
timberwolf
tower
tranquility
troup
upson
vroomfondel
webster
wheeler
william-blount
william-ellery
william-few
william-hooper
william-livingston
william-samuel
william-williams
windstone
wowbagger
yooden-vranx
zarniwoop
zarquon
zem

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 6th, 2014: 1:23AM PDT: All web servers are back online other than the ‘vroomfondel’ web server which is being restored to Debian due to latent hardware issues.  Baratheon, vroomfondel and zem will not be upgraded tonight.

Update October 6th, 2014: 1:00AM PDT: All web servers are now back online other than the ‘cloverleaf’ web server, which should be complete in about 5 minutes. The following servers were not upgraded tonight:

baratheon

vroomfondel

zem

Update October 6th, 2014: 12:33AM PDT: All web servers from batch #3 have been back online for some time. Some web servers from batches #1 and #2 have failed initial tests and will not be upgraded. They will be evaluated for future upgrades, or will be offloaded.  This post will be updated with those servers shortly.
Update October 5th, 2014, 11:36PM PDT: All but 12 servers have completed the upgrade to Ubuntu. They are all in the final stage and should be online shortly. The web server ‘zem’ will not be upgraded tonight, as it did not pass initial tests. It’s back online running Debian now, and we’ll look into the cause tomorrow.

Update October 5th, 2014, 10:42PM PDT: The upgrades are nearing completion. At this time we have 6 batch 1 web servers, 11 in batch 2 web servers and 10 batch 3 web servers which have yet to complete.

Update October 5th, 2014, 10:21PM PDT: Sites from all three batches are continuing to come back online at this time. We will update this post when everything has completed.

Update October 5th, 2014, 10:07PM PDT: The third batch of servers has completed the OS installation phase, and is now running final configurations. Web servers from the first and second batch are coming online now.

Update October 5th, 2014, 9:35PM PDT: The third batch of servers has completed the backup phase, and will be beginning the OS installation momentarily. The second batch of web servers is beginning final configurations. Sites from the first batch of web servers are continuing to come online.

Update October 5th, 2014, 9:10PM PDT: The third and final batch of web servers has begun the backup phase. The second batch of web servers is nearing completion of the  OS installation phase. Sites from the first batch of web servers are continuing to come online.

Update October 5th, 2014, 8:48PM PDT: The second batch of web servers is still in the OS installation phase. Sites from the first batch of web servers are now beginning to come online.

Update October 5th, 2014, 8:40PM PDT:  The second batch of web servers have completed the backup phase and will be beginning the OS installation stage. The first batch is nearing completion of the OS installation stage.

Update October 5th, 2014, 8:35PM PDT:  The second batch of web servers has begun the backup phase. The first batch is nearing completion of the OS installation stage.

Update October 5th, 2014, 8:12PM PDT:  The first batch of 52 webservers has completed the backup phase, and is now entering the OS installation stage.

Update October 5th, 2014, 8:04PM PDT: The first batch of 52 web servers is being backed up now. Once this step is complete the servers will reboot into Ubuntu for OS installation. We’ll be spreading the 5 servers which were not upgraded last night amongst the three batches being upgraded tonight. Those are ‘lupine’, ‘fountaingrass’, ‘oak’, ‘sassafrass’, and ‘gunnera’.

This entry was posted on Tuesday, September 30th, 2014 at 11:31 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

23 Responses to “Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 5th, 8PM – 1AM PDT (RESOLVED)”

My website is down:

We’ve had some trouble reaching citynetevents.com.
Specifically, the error our servers are giving us is: “Operation timed out after 45661 milliseconds with 0 out of -1 bytes received

I am on DECKHAND webserver

@Thomas: I’m sorry to hear that. We’re not upgrading the ‘deckhand’ web server tonight. I’d recommend contacting our LiveChat staff from within the panel. They’ll be happy to assist!

Websites down and its costing me $30-40 in revenue every hour … not cool dreamhost should have started pulling the servers after midnight :/

@Mike: I’m very sorry for the inconvenience. Your web server is in the third batch, which will start coming back online shortly.

@Brandon Thanks for the update

@Mike: No problem! Apaches are starting on your web server now, and your site should be back online momentarily!

My website is on the sundance webserver. Would you mind proving an ETA on when it will be back up and running again?
The down time has been longer then 40min.

@Max, Sorry for the delay, ‘sundance’ took a bit longer than expected. It’s running the final configuration phase now and should be back online within 15 minutes.

My site http://www.antar.in is down from past one hour.I have contacted the customer support they are they can’t do anything as the upgradation work is going on. Please get my site ASAP as i loosing serious amount of money and customers bcoz of your upgradation.Please fix it ASAP..

Thank you for the update @Brandon.

@Max, you’re very welcome. Thank you for your patience!

@Brandon, when can i expect my site to be live.

@Srinivas: Your web server is in the final stage of configurations and will be back online within 15 minutes. I’m very sorry for the delay.

@brandon thanks for the update

@Srinivas: You’re very welcome! Sorry I missed your first message, and thank you for your patience. :)

My website is on Reliant.
Php is running but mySQL still down.

My site is on cloverleaf (http://innovationsinmedicine.eu/), currently not responding. Please advise, when will it be available? I’m really disappointed it takes so long, I’m really losing money with every moment.

Im on vroomfondel still not up ?

@Kenny: Sounds like you’re running a custom PHP.ini that is pointing to an outdated MySQL library. Please remove your php.ini or contact Support for assistance.

@Peter: Sites on cloverleaf have just finished coming back online. I apologize for the delay.

@luke: vroomfondel is currently being restored back into Debian due to latent hardware issues. I’ll update this post as soon as I have further information.

my site is down my server is thomas-mckean please tell me how many hours you need to fix that or i will have to cancel my account ..

Sorry but I’m not using any customm php.ini.
I must switch all my PHP ver. from 5.3 to 5.4.
After that, they came back.

:D

@enalla: thomas-mckean has been online for some time. I’m also able to load your site without error.

@Kenny: Thanks for the info!

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 4th, 8PM – 1AM PDT (RESOLVED)
11:26 am

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 4th, 8PM – 1AM PDT (RESOLVED)

Posted (September 30th, 2014 at 11:26 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 4th. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

agapanthus
agba
ajuga
anemone
araucaria
ash
aster
astilbe
ayan
balsa
baptisia
barrenwort
basswood
bergenia
bloodroot
bloodwood
bluestar
boltonia
boxelder
boxwood
brazilwood
bubinga
buckeye
butternut
campion
candytuft
carapa
catmint
cattail
centaurea
chives
clematis
cocobolo
coleus
coneflower
coralbells
corkwood
cottonwood
cucumbertree
cypress
delphinium
dogwood
elm
eucalyptus
evolvulus
fennel
fir
fleabane
foamflower
fountaingrass
foxglove
fuchsia
gaura
gazania
gentian
geranium
goatsbeard
goldenrod
goldenstar
greenheart
grenadilla
gum
gunnera
hackberry
helenium
hellebore
hemlock
hibiscus
honesty
hosta
ipe
iroko
ironwood
jacaranda
kiwi
lantana
lavender
lemongrass
liverleaf
lupine
malva
maple
melon
merbau
mpingo
muscida
navi
nekkar
neshmet
nunki
oak
padauk
palisander
peacock
phad
pleione
procyon
propus
proxima-centauri
pulcherrima
ramin
rasalas
rastaban
redcedar
redheart
rigel
rotanev
ruchba
rukbat
sabik
sadachbia
sadalbari
sadalsuud
sadr
sal
salm
sandalwood
santamaria
sargas
sarin
sassafras
sceptrum
scheddi
schedir
seginus
sham
sheratan
situla
spica
spruce
subra
sulafat
syrma
tabit
talitha-australis
talitha-borealis
tambotie
tarazet
taygeta
teak
tegmen
terebellum
unuk
ushakaron
vindemiatrix
walnut
wasat
wazn
wezen
zuben-el-akrab

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 5th, 2014, 12:11AM PDT: The upgrades have successfully completed. The following servers will be re-scheduled as soon as possible:

 

lupine, fountaingrass, oak, sassafrass, and gunnera

Update October 5th, 2014, 12:02AM PDT: The Ubuntu upgrades for tonight have completed. The following servers will be upgraded at a later date:

lupine, fountaingrass, oak, sassafrass, and gunnera

Update October 4th, 2014, 11:52PM PDT: All web server upgrades have completed other than ‘spruce’ which should be done momentarily. We are now testing to ensure everything is running properly.  The following web servers have had their Ubuntu upgrades postponed:

lupine, fountaingrass, oak, sassafrass, and gunnera

Update October 4th, 2014, 11:22PM PDT: Batch #1 & #2 have completed. 5 servers remain from batch #3. After all batches have completed we’ll begin testing to ensure everything is in order.

Update October 4th, 2014, 11:08PM PDT: Batch #1 has completed, and all sites are online. Batch #2 has two servers left to complete which should be done shortly. Batch #3 has 6 servers to go, and will then be complete.

Update October 4th, 2014, 10:37PM PDT: Sites from the third batch are coming back online now. The second batch is complete other than 5 servers and the first batch is all online except for 3 servers.

Update October 4th, 2014, 10:25PM PDT: The third batch has entered the restoration phase. The second batch is complete other than 5 servers, and the first batch has 3 servers to finish.

Update October 4th, 2014, 10:05PM PDT: The third batch has entered the OS installation phase. The second batch is still restoring and most servers from the first batch is now all online other than a few stragglers.

Update October 4th, 2014, 9:50PM PDT: The third and final batch of webserver has now started the backup phase. The second batch is now restoring and should be back online shortly. The first batch is now back online other than a couple of stragglers which should be up shortly.

Update October 4th, 2014, 9:45PM PDT: The second batch of servers is now running initial configurations. The first batch is 90% back online.

Update October 4th, 2014, 9:35PM PDT: Most of the servers from the first batch are now back online. The second batch is still in the OS installation stage.

Update October 4th, 2014, 9:15PM PDT: The second batch of servers has finished backing up and is botting into Ubuntu for OS installation. The first batch is completing configurations and should be back online shortly.

Update October 4th, 2014, 9:07 PM PDT: The second batch of servers has now begun the backup stage. The first batch is completing configurations and should be back online shortly.

Update October 4th, 2014, 9:00 PM PDT: The first 50 web servers have completed OS installation and initial configurations are now running.

Update October 4th, 2014, 8:30 PM PDT: The first 50 web servers have finished backing up, and are now booting into Ubuntu for OS installation.

Update October 4th, 2014, 8:08 PM PDT: We are now beginning the upgrade to Ubuntu. We’ll be running these upgrades in 3 separate batches 50 machines. This stage consists of backing up all existing configurations and data.

This entry was posted on Tuesday, September 30th, 2014 at 11:26 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading 100 shared web hosting servers to Ubuntu 12.04 Precise – September 28th, 8PM – 1AM PDT (RESOLVED)
9:26 am

Upgrading 100 shared web hosting servers to Ubuntu 12.04 Precise – September 28th, 8PM – 1AM PDT (RESOLVED)

Posted (September 22nd, 2014 at 9:26 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 100 web servers this week. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of this time will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

ailsabay
aken
alkan
amun
amunet
anhur
anuket
bach
ballechin
balvenie
bartok
berlioz
berwald
bes
brahms
brown
bunnahabhain
buri
caolila
cardhu
chopin
clementi
clynelish
copland
daftmill
dailuaine
demeij
dvorak
eir
eisler
el-dabh
elli
eostre
fitch
freya
frigg
fulla
gefjun
glenburgie
gliere
gorecki
haydn
henet
hlin
honegger
horus
hummel
inchgower
iusaaset
ivanovs
ives
kaelin
karlowicz
kashif
kebechet
khepri
koechlin
koppel
lagavulin
lange-muller
liszt
loch-lomond
lofn
mani
martinu
mccabe
mendelssohn
menezes
menhit
milhaud
nanna
north-british
nott
pakhet
panufnik
parry
part
port-charlotte
prokofiev
qebui
renenutet
rontgen
rubinstein
sallinen
schumann
shapiro
shostakovich
speyburn
strathearn
tchaikovsky
teaninich
the-speyside
tomatin
tomintoul
tormore
villa-lobos
whear
willson
wingate
witt

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 28th, 2014, 3:11AM PDT:  Alright DreamHosters, thanks for staying up with us tonight! We’ve finished testing and are calling this all clear! If you run into any issues on your sites that you believe may be due to this upgrade, please contact our Technical Support team. They’ll be happy to help!

Update September 28th, 2014, 2:15AM PDT:  Our systems engineers are finishing up testing, and we should have the all clear soon.

Update September 28th, 2014, 1:25AM PDT:  Apache configurations are still running for those that aren’t yet online, and we’re continuing to run test suites to ensure everything is running as it should be.

Update September 28th, 2014, 12:43AM PDT: There are a handful of Apache services that haven’t come online properly due to issues binding with ipv6 addresses. These services are being configured now and should be running properly shortly.

Update September 28th, 2014, 12:25AM PDT: Sites are now coming online for  tormore, koechlin, panufnik, and fulla. All other webservers should now be fully operational and post-upgrade tests are executing.

Update September 28th, 2014, 11:49PM PDT: Sites should now all be online except for those hosted on tormore, koechlin, panufnik, and fulla. These machines should also be complete soon.

Update September 28th, 2014, 11:33PM PDT: Post-install configurations have completed, and sites are coming online now.

Update September 28th, 2014, 10:43PM PDT: This step is taking a bit longer than expected, but things are still progressing smoothly.

Update September 28th, 2014, 10:07PM PDT: Post-install configurations are wrapping up, and sites should begin coming back online shortly.

Update September 28th, 2014, 9:51PM PDT: Ubuntu OS installation has completed on all machines, package installation has finished on most, and post-install configurations have begun.

Update September 28th, 2014, 9:36PM PDT: Ubuntu OS installation has completed on most machines, and package installation has begun.

Update September 28th, 2014, 8:44PM PDT: The backup phase has completed successfully, and the web server are now being rebooted into the Ubuntu OS.

Update September 28th, 2014, 8:30PM PDT: We’ve started the upgrade, and are backing up configurations before rebooting into Ubuntu.

Update September 28th, 2014, 8:09PM PDT: We’ve started final pre-upgrade checks for the listed web servers.

This entry was posted on Monday, September 22nd, 2014 at 9:26 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

36 Responses to “Upgrading 100 shared web hosting servers to Ubuntu 12.04 Precise – September 28th, 8PM – 1AM PDT (RESOLVED)”

my server still down :(
‘schumann’

thx …… my server in nw up :)

@Happy SIngh: Glad to hear it! :)

port-charlotte, not seeing web services up yet

I’ve been using PHP from cron, and the path seems to have changed. I was using /usr/local/php5/bin/php (from http://wiki.dreamhost.com/Run_PHP_from_cron). What is the correct path now?

@Matt: I’m seeing web services active on port-charlotte. Please reply with your domain name so I can further investigate.

@Øyvind: PHP5.2 is not available on Ubuntu, you can use the 5.3 binary: /usr/local/php53/bin/php

satinjazz.com

My sites are down and they are hosted on scheddi, is it related to the update?

@Matt: Thank you, I’m talking with our engineers now.

@Matt: Your Apache service had an issue binding to an ipv6 address and should be up momentarily. :)

@Pacy: Sorry about your site! This isn’t related to the Ubuntu upgrade, please contact our support team via ticket or livechat and they should be able to help you.

Good now thanks Brandon

I’m on copland and cannot log in using FTP. Is this a known issue?

@Matt: Glad to hear it!

@Dale: Thanks for the info, checking into it now.

None of my sites on gefjun are working either. I’m just getting a timeout connecting to port 80 on 75.119.221.67.

@Dale: Looks like it’s to be expected. The engineers top priority is getting your sites back online, so secondary services are scheduled to be brought back online afterward. FTP should be back online soon, thanks for your patience!

@Phil: Pushing through your Apache configuration now, should be up shortly. I apologize for the delay.

Hi! My site on “ailsabay” is not working (I have just check it).

Everything is working now. Thanks Brandon.

@JPinto: My apologies for the trouble, it looks like there are some Apache services not properly binding to ipv6 addresses, your site is now online, and I’ve updated the post to reflect these issues.

@Phil: Very glad to hear it. Thank you for your patience.

Thanks for the quick response, Brandon. All good now – FTP is back :)

@Dale: Nice! You’re very welcome.

Thank you very much for your kind and quickly answer. Site´s on!

@JPinto: You’re welcome! I’m happy to hear it. :)

my server still down – ‘honegger’

My server is not on the list (glendullan), but it is unavailable too. Got the bulk response from tech support that the maintenance is running. I hope it will be back soon?

@Selvan: Sorry to hear that. Your Apache is being re-configured now, and your sites should be up shortly.

@Bart: Sorry for the confusion! Your webserver was not upgraded, and your sites are now online.

Thanks….’honegger’ is working fine and all sites are online

@Selvan: Great! Thanks for the update. :)

Hi! My web site on “bartok” is not working

@michael: The Apache service for your domain is configuring now, and your site should be up shortly. I apologize for the delay.

“Thanks….’honegger’ is working fine and all sites are online”

not really. all my sites are down

@juen: Please let me know an affected domain so I can troubleshoot this for you.

My sites are down, they’re on kaelin

unit3.net
cooloolatrailcarealliance.org.au

@Danny: Thanks for letting me know, I apologize for the extended downtime. We’re working our way through the list of affected services, and are almost done. Your Apache service is being configured right now and your sites will be back online momentarily.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading seventeen shared web hosting servers to Ubuntu 12.04 Precise – September 14th, 9PM – 12AM PDT (RESOLVED)
4:26 pm

Upgrading seventeen shared web hosting servers to Ubuntu 12.04 Precise – September 14th, 9PM – 12AM PDT (RESOLVED)

Posted (September 12th, 2014 at 4:26 pm PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

Upgrading to Ubuntu from Debian will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve upgraded six web hosting machines to Ubuntu so far, and things have gone swimmingly.

We’re now going to upgrade seventeen additional shared web servers:

applewood
artemisia
avens
bluebeard
bunchberry
burnet
columbine
coreopsis
deadnettle
dianthus
flax
globeflower
lovegrass
meranti
mint
pherkad
wenge

 

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 14th, 2014, 11:45 PM PDT: We’ve verified that everything is running properly, and we’re now calling this complete!

Update September 14th, 2014, 10:30 PM PDT: All services are now back online, and we’re now double checking to ensure everything is in order.

Update September 14th, 2014, 10:24 PM PDT: Sites are now coming back online, and the upgrades should be completed momentarily.

Update September 14th, 2014, 10:17 PM PDT: We hit a small snag during  final configurations, which has now been figured out. Final configurations are resuming and we should be finished soon!

Update September 14th, 2014, 9:40 PM PDT: Package installation has completed, and configurations are running now.

Update September 14th, 2014, 9:26 PM PDT: The OS have finished installing and packages are now being installed.

Update September 14th, 2014, 9:21 PM PDT: The servers have booted into Ubuntu and the OS is now installing.

Update September 14th, 2014, 9:16 PM PDT: We are now beginning the upgrade for the aforementioned web servers.

This entry was posted on Friday, September 12th, 2014 at 4:26 pm and is filed under General Outages, Multiple Server Issue, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

12 Responses to “Upgrading seventeen shared web hosting servers to Ubuntu 12.04 Precise – September 14th, 9PM – 12AM PDT (RESOLVED)”

Does this change the RSA key fingerprint as well? The fingerprint is not matching that displayed in the panel any more.

@bob: The RSA key fingerprint will change, however the panel should update with the new key once the upgrade has completed.

Seems to be taking a long time. Is anything wrong?

@bob The syseng team hit a snag during the final configurations, but we’re now back on track. Sorry that this is taking longer than planned!

@Brandon, ok, understood. Are we now looking at an ETA of minutes (hopefully) or hours?

@bob Minutes! Sites are now coming back online, and once we’ve verified all servers are 100% complete, we’ll begin testing to ensure everything is perfect.

@Brandon, great news. I’ve just seen mine coming back up. Great job! Thanks. I look forward to my server and home computers sharing the same OS!

@bob, very glad to hear it! Would you mind verifying that your RSA key in the panel is correct now? If not, I can look up your account if you provide your domain name.

@Brandon After a slight delay (3-4 minutes), the panel is now showing the same key as the one I see when I try to ssh to coreopsis. It looks like everything is well so far!

@bob wonderful, thanks for the update!

Our site is still down?

@Michael I’m seeing your domain as online. It is however a bit slow, and appears to be getting a ton of traffic. I’ve enabled wp-super-cache for you, but if you continue to see performance issues, you may wish to submit a support ticket and request optimization assistance.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 4.0 Major Release Now Available
10:47 am

WordPress 4.0 Major Release Now Available

Posted (September 4th, 2014 at 10:47 am PST) by Mika E.

Baboom!

Can you believe we’re on 4.0 already?

For all the wicked details, check out the official WordPress release notes for 4.0 “Benny”. I’m in love with the new media library gallery! Getting to see all my images like that makes it easier for me to see if I’ve already uploaded that cool picture of my cat as a beard before I upload it again for a new post. If you’re one of those people who adored the old way, though, and had extra columns of joy, don’t worry. You can switch right over and change your view. Also the way the post editor sizes itself to match my screen has made it easier to write epic long posts about how my cat hates being a beard, and that’s how I got this scar on my face.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

By the way, WordPress uses semantic versioning, so in WP speak, 3.9 and 4.0 are the same level of absolute super-awesome. Don’t let that scary point-oh version number freak you out; we’re WordPress, not Windows.

If you run into issues after your upgrade, remember to report any problems you have with specific plugins and themes to those developers on WordPress.org (or the dev’s website)! The WordPress core devs work really hard to make sure everyone tests, but you know how real life gets in the way of real life? It happens.

This entry was posted on Thursday, September 4th, 2014 at 10:47 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

7 Responses to “WordPress 4.0 Major Release Now Available”

Not soon after I upgraded to WP 4.0, my website was hacked. Not sure how to handle.

Since the Word Press 4.0 update yesterday I can not do many of the NEEDED functions to update my Corona theme from Theme Forest. I’ve contacted them but am pressed to get going so hope that maybe you guys know about these issues.

I can no longer access my pictures as I try to upload a featured media, the button does not click, also in the Corona themes under template builder all my template list is gone and can not even create a new one. Not sure if this is something word press can fix or if it is Kriesi that needs to do it.

Thanks for any help.

Could it be the database update?

Since the Word Press 4.0 update yesterday I can not do many of the NEEDED functions to update my Corona theme from Theme Forest. I’ve contacted them but am pressed to get going so hope that maybe you guys know about these issues.

I can no longer access my pictures as I try to upload a featured media, the button does not click, also in the Corona themes under template builder all my template list is gone and can not even create a new one. Not sure if this is something word press can fix or if it is Kriesi that needs to do it.

Thanks for any help.

Could it be the database update?

Corona theme from Theme Forest. Kriesi

Thanks for any help.

X160e7 hi!,I like your writing so much! share we communicate more about your article on AOL? I require a specialist on this area to solve my problem. May be that’s you! Looking forward to see you.

Thank you for your blog post.Much thanks again. Great.

I appreciate you sharing this article post. Really Great.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading five shared web hosting servers to Ubuntu 12.04 Precise – September 4th, 9PM – 12AM PDT(RESOLVED)
10:58 am

Upgrading five shared web hosting servers to Ubuntu 12.04 Precise – September 4th, 9PM – 12AM PDT(RESOLVED)

Posted (September 3rd, 2014 at 10:58 am PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

The main reason for upgrading from Debian to Ubuntu is that it will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve already completed the upgrades on some backend machines including  VPS host machines, mail servers and the shared web server ‘daugherty’.  As the upgrade for ‘daugherty’ went as planned, we’re now going to upgrade five additional shared web servers:

 

whale
talisker
laphroaig
nephthys
scriabin

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 4th, 2014, 10:35 PM PDT: We’ve verified that everything is running properly, and we’re now calling this complete!

Update September 4th, 2014, 9:50 PM PDT: All sites are online, and we’re now double checking to ensure everything is in order.

Update September 4th, 2014, 9:50 PM PDT: Sites are now coming back online and the upgrades should be complete momentarily.

Update September 4th, 2014, 9:20 PM PDT: Package installation has completed, and configurations are running now.

Update September 4th, 2014, 9:20 PM PDT: The machines are now booted into Ubuntu and installing packages.

Update September 4th, 2014, 9:12 PM PDT: The machines are now actively installing the Ubuntu OS.

Update September 4th, 2014, 9:03 PM PDT: We’ve started the upgrade and will provide further updates as it progresses.

This entry was posted on Wednesday, September 3rd, 2014 at 10:58 am and is filed under Announcement, Multiple Server Issue, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Connectivity Issues (RESOLVED)
12:17 am

DreamObjects Connectivity Issues (RESOLVED)

Posted (August 29th, 2014 at 12:17 am PST) by Steven

Our cloud engineering team is investigating connectivity issues being reported by the DreamObjects service. We do not have an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe and we will continue to update this post as we gather more information.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update August 29th, 2014, 02:15 AM PDT: All tests have completed successfully! Please contact support if you still see any issues with the DreamObject service.

Update August 29th, 2014, 02:15 AM PDT: DreamObjects connectivity has been restored! Tests are still running, though, so we can be sure there are no remaining issues.

Update August 29th, 2014, 02:06 AM PDT: Work continues to resolve the problem.

Update August 29th, 2014, 01:06 AM PDT: There isn’t anything new to report at this time. Work continues to get this fixed as quickly as possible. We thank you for your patience!

Update August 29th, 2014, 12:45 AM PDT: The trouble appears to be a cluster reporting it is full when it is not. We are working to resolve this but still no ETA as of this moment.

This entry was posted on Friday, August 29th, 2014 at 12:17 am and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Intermittent Delays ( RESOLVED )
10:34 am

DreamObjects Intermittent Delays ( RESOLVED )

Posted (August 26th, 2014 at 10:34 am PST) by Justin@DH

DreamObjects is experiencing higher than normal traffic and experiencing intermittent delays in processing requests. Our Cloud Engineering team is continuing to monitor the situation and make adjustments as needed. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact our support team for assistance.

Update August 26th, 2014, 3:57 PM PDT: DreamObjects service has been stable for the last 4 1/2 hours. Our Cloud Engineering team will continue their routine monitoring for any further traffic spikes.

Update August 26th, 2014, 11:30 AM PDT: Traffic levels have stabilized to within normal limits and we’re no longer seeing intermittent delays. Our Cloud Engineering team will continue monitoring traffic levels and ensure the cluster remains responsive.

This entry was posted on Tuesday, August 26th, 2014 at 10:34 am and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “DreamObjects Intermittent Delays ( RESOLVED )”

Working on the new site today, seems objects are still down and dreamhost panel inaccessible.

Working on the new site today, seems objects are still down and dreamhost panel inaccessible. and just trying to post this throws an error! haha what the heck?

Error 503 backend read error

backend read error

Guru Mediation:

Details: cache-at52-ATL 1409238885 3985879100

Varnish cache server

Seems like some of dreamhost’s own sites are down! There is something going on over at dreamhost…

All of my DreamObjects are down, not sure what is going on. I use DO for backups, but missing those now. It is midnight on the 29th..

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading shared web hosting server ‘daugherty’ to Ubuntu 12.04 Precise – August 27th, 9PM – 12AM PDT(RESOLVED)
4:00 pm

Upgrading shared web hosting server ‘daugherty’ to Ubuntu 12.04 Precise – August 27th, 9PM – 12AM PDT(RESOLVED)

Posted (August 25th, 2014 at 4:00 pm PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

The main reason for upgrading from Debian to Ubuntu is that it will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve already completed the upgrades on some backend machines including  VPS host machines and mail servers, but we’re not taking this upgrade for granted. In order to ensure things go as smoothly as possible, we’ll be starting this upgrade on a single server, ‘daugherty’.

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re on the ‘daugherty’ web server by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update August 27th, 2014, 10:05 PDT:  The upgrade of ‘daugherty’ to Ubuntu has completed without issue. Sites are now back online. If you’re seeing any issues with your sites, please don’t hesitate to contact our support team.

Update August 27th, 2014, 9:50PM PDT:  Final configurations have been completed, and we’re now running checks on all running services. Sites will be coming online momentarily.

Update August 27th, 2014, 9:37PM PDT:  Configurations have run and services are now being restored. Next will be final configurations and the upgrade will be complete.

Update August 27th, 2014, 9:27PM PDT:  All packages have been installed, updates are running and configuration management will launch shortly.

Update August 27th, 2014, 9:12PM PDT:  The team has completed creating backups of all long-running content and is now updating related metadata. The server is now booted into Ubuntu and installing packages.

Update August 27th, 2014, 9:00 PM PDT: We’ve started the upgrade and will provide further updates as it progresses.

This entry was posted on Monday, August 25th, 2014 at 4:00 pm and is filed under Announcement, Single Server Issue, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Tuesday, August 26th, 12am to Wednesday, August 27th, 12am PDT
5:40 pm

Live Chat Unavailable Tuesday, August 26th, 12am to Wednesday, August 27th, 12am PDT

Posted (August 21st, 2014 at 5:40 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Tuesday, August 26th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in our quarterly team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 12am PDT the next morning, August 27th. Thank you for your patience and understanding.

This entry was posted on Thursday, August 21st, 2014 at 5:40 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Network Maintenance in US-West (Los Angeles) Data Center – August 21st 10pm PDT (4 hrs) (RESOLVED)
1:35 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – August 21st 10pm PDT (4 hrs) (RESOLVED)

Posted (August 20th, 2014 at 1:35 pm PST) by Brian@dreamhost

We will be conducting network maintenance in our US-West (Los Angeles) Data Center starting at 10pm PDT. This is to perform routine maintenance on some of our network hardware. We estimate the maintenance window to be 4hrs, during which we will move connectivity from one piece of hardware to a temporary piece, then perform the maintenance, then move back. Each of these moves may cause up to 30 second of service pause, and we will be closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the MySQL (shared or VPS) machines are located. In order to check if you will be affected, please check DreamHost panel for the location of your MySQL server. If you host dynamic web sites with us and your mysql server is at the Los Angeles location, those may also be affected indirectly, but there will be no impact on static sites or email service during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update August 21st, 2014, 10:41 PM PDT: All mysql machines have been moved back to their starting router, and are now functioning correctly. We are concluding this maintenance as of now!

Update August 21st, 2014, 10:34 PM PDT: Moving each machine, one by one still. Each move is checked for issues and verified to be working. So far we have no seen any issues.

Update August 21st, 2014, 10:05 PM PDT: We are now currently moving each mysql machine over, and monitoring each move closely.

Update August 21st, 2014, 9:58 PM PDT: We are here and setting up. We’ll update this post when we’re ready to begin

This entry was posted on Wednesday, August 20th, 2014 at 1:35 pm and is filed under Announcement, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Intermittent Connectivity Issues (RESOLVED)
10:23 am

DreamObjects Intermittent Connectivity Issues (RESOLVED)

Posted (August 13th, 2014 at 10:23 am PST) by Justin@DH

Our cloud engineering team is investigating intermittent connectivity issues being reported by the DreamObjects service. There is not yet an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

This entry was posted on Wednesday, August 13th, 2014 at 10:23 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “DreamObjects Intermittent Connectivity Issues (RESOLVED)”

Not resolved as your subject line indicates. “DreamObjects Intermittent Connectivity Issues (RESOLVED)”

@Ray: We are continuing to monitor DreamObjects, and from our tests it appears to be working normally. If you are still seeing issues please contact support with details and we would be happy to look.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.9.2 – Security Release
1:46 pm

WordPress 3.9.2 – Security Release

Posted (August 6th, 2014 at 1:46 pm PST) by Mika E.

WordPress 3.9.2 is now available as a security release for all previous versions of WordPress, and is being applied now to all sites who allow for Background Updates in WordPress, as well as any who use our Auto Update via Panel.

You can read the release notes for 3.9.2, and this release is coordinated with a Drupal release, as both projects share a potential vulnerability (due to an issue in PHP) that could cause a denial of service attack.

As a reminder, WordPress itself performs background updates of all minor releases, since WordPress 3.7 was released in 2013. That means even if you have disabled auto updates via panel (which you’d never do, because updating is the best thing since sliced bread), WordPress will still upgrade your 3.7, 3.8, and 3.9 installs to the latest, most secure, version.

If for whatever reason you went and disabled all possible update methods to your site, please log in to your WordPress site now, head to Dashboard → Updates and simply click “Update Now.” It’s just that easy.

Speaking of updates …. WordPress 4.0 is due out by the end of August! Now’s the time to have a look at your sites and make sure all your plugins and themes are up-to-date in preparation for some cool new features. You may even want to upgrade your PHP settings in panel to PHP 5.4, just to be extra cool.

This entry was posted on Wednesday, August 6th, 2014 at 1:46 pm and is filed under Announcement, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “WordPress 3.9.2 – Security Release”

You know what would really be extra cool? Giving us the ability to upgrade to PHP 5.5. It came out of beta over a year ago, and 5.4 has less than a year of support left.

August 13th, 2014 at 7:13 amTiago Nicastro Says:

WEBMAIL DOWN! WEBMAIL DOWN! WEBMAIL DOWN! WEBMAIL DOWN!

Email is having a 20 minute delay.

unfortunately almost all the plugins are not even compatible with the 3.9.2 and the sites I did update are now having memory leak errors and I have no way of backing them back down the previous version because it updates the install package automagically. So glad that was a test site. So glad the autoupdates are turned off. Waiting patiently as they fix the issue.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 24th 10pm PDT (4 hrs) (RESOLVED)
4:28 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 24th 10pm PDT (4 hrs) (RESOLVED)

Posted (July 21st, 2014 at 4:28 pm PST) by andrea

Following in the path of the successful network maintenance that increased network redundancy and capacity for the MySQL servers in our US-West (Los Angeles) data center, our next step is to repeat the process for our shared web servers in that same location.

The maintenance will happen on July 24th, starting at 10pm PDT. We estimate the maintenance window to be 4hrs, during which we will be moving groups of servers from the old network uplink switch to the new ones. Each of these moves may cause up to 30 second of service pause, and we will be very closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the shared web servers are located. In order to see if you will be affected, please see check the DreamHost panel for the location of your shared web server. There will be no impact on MySQL or email services during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update July 24th, 2014, 11:44 PM PDT: We are happy to report that the maintenance completed successfully, way ahead of schedule and without noticeable service impact. We are closing the comments, but if you have any questions or need any assistance, please let us know by contacting tech support.


Update July 24th, 2014, 11:44 PM PDT: The maintenance has been proceeding with minimal or unnoticeable effect on our services. We appreciate your continued patience while we work towards completion.

Update July 24th, 2014, 10:35 PM PDT: Just a quick update to let you know that we’re going through the steps as scheduled, so far with no impact at all. We’ll update this post again in about an hour unless there is a status change.

Update July 24th, 2014, 10 PM PDT: We are commencing the maintenance as scheduled. We’re opening comments for the duration, and we’ll update this post as we’re progressing through the steps. We appreciate your patience and understanding while we are working on this.

This entry was posted on Monday, July 21st, 2014 at 4:28 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 24th 10pm PDT (4 hrs) (RESOLVED)”

I am not able to access all of my sites. I am not able to access even control panel(panel.dreamhost.com). Is this due to the network maintainance? How do I know that?

@mpchekuri – that sounds unrelated, because when your comment was posted, we didn’t touch anything customer-facing yet. Sorry about the delay in getting back to you, but if you give me one of your domain names, I can look into what is causing your issue – if you’re still experiencing it.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Service Interruption (RESOLVED)
4:22 am

Live Chat Service Interruption (RESOLVED)

Posted (July 20th, 2014 at 4:22 am PST) by Steven

It appears the provider for our Live Chat service is experiencing trouble at this time. The service has been up and down over the last couple of hours. Please be aware that if you do connect with a tech. support representative it is possible you may be disconnected (it’s happening on our end, too). Thank you for your patience and understanding.

Update July 20th, 2014, 5:10 AM PDT: We have confirmed with LivePerson that the issue was indeed on their end and has been resolved. We do not anticipate any further trouble with the chat service but definitely let us know if you encounter any errors!

This entry was posted on Sunday, July 20th, 2014 at 4:22 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Live Chat Service Interruption (RESOLVED)”

My site is still down!!!

Live chat does not work at all!

great! now just wait 30 minutes on hold until someone answers…

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues for subset of customers in US-West Data Center (RESOLVED)
12:31 am

Connectivity Issues for subset of customers in US-West Data Center (RESOLVED)

Posted (July 19th, 2014 at 12:31 am PST) by Steven

We are currently experiencing some power related issues with one of our racks in our Los Angeles data center. We apologize for the trouble and are working as quickly as possible to restore service. The servers impacted by this are:

hindemith, holmboe, honegger, hovhaness, hummel, dindy, ivanovs, ives, kaelin, karlowicz, kashif, koechlin, koppel, lalo
liszt, lange-muller, martinu, mahler, melartin, mccabe, mendelssohn, menezes, messiaen, mignone, milhaud, mozart
nielsen, nystroem, panufnik, parry, part, penderecki, persichetti, prokofiev, delpuerto

You can lookup which server you are hosted on in the admin panel by clicking the Account Status link at the top-right.

Watch for additional updates over the next half to one hour. If you have any questions or need anything else please Contact Support. Thank you!

Update July 19th, 2014, 1:40 AM PDT: This is resolved now and all servers are up. Once again, we thank you for your patience. If you still see any problems with your service please contact support through the link above. Thanks!

Update July 19th, 2014, 1:11 AM PDT: We are still working on the problem. Unfortunately, we are unable to provide an ETA for resolution at this time. Thank you for your patience and we will provide another update as soon as possible.

This entry was posted on Saturday, July 19th, 2014 at 12:31 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

5 Responses to “Connectivity Issues for subset of customers in US-West Data Center (RESOLVED)”

I still got problems… mysql database errors… someone else too?

My sites on dogwood are not working…

Actually dogwood sites are finally responding.. but extremely slow.

thomas-fitzsimmons also appears to be affected.

I am out!!! it says:[Fiddler] The socket connection to mumblebeeinc.com failed.
ErrorCode: 10060.
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

I am in Illinois….

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Limited service availability at our US-West (Los Angeles) Data Center (RESOLVED)
11:59 am

Limited service availability at our US-West (Los Angeles) Data Center (RESOLVED)

Posted (July 16th, 2014 at 11:59 am PST) by andrea

We are currently experiencing connection issues at sections of our US-West (Los Angeles) data center. Our operations teams are already looking into it, and we’ll update this status post as soon as we have more information. Very sorry about the inconvenience this is causing.

Update July 16th, 2014, 2:32 PM PDT: We are happy to report that all affected services have been fully restored, including mail and shared web servers.
While working on this, we also took steps towards avoiding similar situations in the future: we rerouted the shared web services from the affected panel to a different power source and moved some redundant circuits off as well to further alleviate load on said panel. Again, please accept our apologies for the inconvenience this has caused. If you have any questions or concerns, please contact tech support, and we’ll be happy to help!

Update July 16th, 2014, 2:05 PM PDT: All the affected shared web servers are now using a different power source and are fully operational. Our mail machines are coming back online one after each other. We are getting close to being fully restored and appreciated your continued patience in the meantime.

Update July 16th, 2014, 1:42 PM PDT: We have completed moving most of shared web servers off of the affected power panel. Once these machines boot back up our shared service will be 100% restored and we will be continuing work on restoring mail. As a reminder, this is only affecting a small number of shared web services (and soon none at all) and mail that’s located in our US-West (Los Angeles) data center. No other data centers or MySQL services were affected.

Update July 16th, 2014, 1:20 PM PDT: We were in the process of restoring all affected services when the same panel (see previous update) went down again. We are very sorry about this added inconvenience. Before the panel lost power for the second time, we began rerouting power to other power distribution units as a precaution. We are now in the process of moving services to those new circuits and will update this status post again as soon as we have more information.

Update July 16th, 2014, 12:45 PM PDT: We are happy to report that power is being restored now, and our operations teams are working diligently to ensure that every service is up and running again as expected. The confirmed cause of this service interruption was that a power panel that our setup connects to went down. Because this panel exists in between our machines and our provider’s generators and UPS system, emergency power was unavailable. We are working on correcting this in the future, and thank you for your patience while we make all services available again.

Update July 16th, 2014, 12:28 PM PDT: We are continuing to working on this situation. To repeat this is affecting mail service and a small number of shared web servers in our US-West (Los Angeles) data center. Checking your location in the DreamHost panel will show if you’re affected. Very sorry for the inconvenience!

Update July 16th, 2014, 12:05 PM PDT: We have identified the source of the connection problems: an isolated number of mail machines and shared web servers have lost power in this data center. This is not affecting MySQL services. We’re working together with our provider to restore availability. We will update this post again as soon as we have a status change to report or when we have more information.

This entry was posted on Wednesday, July 16th, 2014 at 11:59 am and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

194 Responses to “Limited service availability at our US-West (Los Angeles) Data Center (RESOLVED)”

Mail server down!

Whiskey Tango Foxtrot!

Mail server down. I can’t afford it to be so any updates ASAP would be great. Thanks.

Is this isolated to homie sub4?

Email down. Website still up. Fix it ASAP please.

July 16th, 2014 at 12:08 pmBakamoichigei Says:

That must be what’s wrong with my webmail…

Why Why oh Why? Not Today

Is there not failover? No UPS(s)? Please bring us up asap…..

Email service absolutely down

Everything down.
Service scan
FTP – 21 Error: TimedOut
SMTP – 25 Error: TimedOut
HTTP – 80 Error: TimedOut
POP3 – 110 Error: TimedOut
IMAP – 143 Error: TimedOut

Auch!
Email is linked to all my webstores. This is driving me and my staff crazy.
please resolve ASAP – WE GOT NO EMAIL SERVICE!

Very sorry about the inconvenience this is causing. We updated the status post to indicate that it’s affecting mail service and some shared web servers in that data center, and we’ll post again as soon as we have more information. We’re working on resolving this as soon as possible.

Does this have anything to do with any anti-spam solution? At about 11 AM Central/9 AM Cali time I started getting a ton of NDR’s and spam. And now my mail is completely down.

Webmail down, User not recognized at login.

It’s not just my e-mail that’s down, but all my websites, as well.

Wow guys – this mail server down in the middle of a Wednesday is just NOT cool.

OH MY GOD!!!!! NEED HELP PEOPLE BECOME CRAZY OUTHERE!!!!

our emails have been having random delays for weeks now, and no one seems to know how to fix it. It happens every other day, and lasts a few hours, but I cannot run our business with these types of issues!

I’ve got webserve and SMPT for outgoing but no incoming mail. Tick tock tick tock. Thanks for fixing it asap!

Email server down.

Is there not 2nd tier power source in place for such an event? Clients need their files out today!

July 16th, 2014 at 12:18 pmMatt Neuburg Says:

Email down. I do appreciate these updates, though.

on homiemail-sub4 dead in the water!!!!

Webmail is down, website is up. When trying to access email, get an error.

This is 2014 correct?

This is the most important day of my life. I cannot believe this is happening. It is ruining the most important day of my life on this, the most important day of my life.

I cant believe there is no redundancy. You only realise what you have bought into when it goes wrong. Power fail is one of the most obvious areas to have back-up. Battery, generator anything would help.
Not Good!

NOw what… all days have a problmes…

How hard is it to keep the mail server running?

I recall this has happened before a year or two ago because someone turned off a power strip to the servers. It sounds like the power provisions are less than adequate there.

homiemail-sub5 affected as well

Every time Dreamhost goes down I take the time to move whatever is left running on Dreamhost elsewhere.

Our mail server is down. We have a staff of 21 not receiving mail right now. Please update us when it’s up. thank you

How widespread is this? I am on the east coast and I got no email…

Mail servers down!!! WTH!

Help people in problems in here!!!

We have no mail !!!

More email problems and issues with DreamHost. What else is new…

My site is down. But it’s not the end of the world. I understand that budget, shared servers are not for 99.99999% critical applications, unlike so many people here who apparently run their businesses off them.

Bummer, but thanks for keeping us updated. Hope it’s a quick fix!

whiskey tango foxtrot all mail servers down

It’s not even possible to submit an official report to DH because I CANNOT access the web panel because that requires that I authenticate my IP address, but to do that requires the mail server for me to do the authentication, but the mail server is down!

So there is not a way to contact DH staff officially.

We would like to apologize again for the downtime this is causing. Just to reiterate, mail service and some shared web service is affected in our US-West (Los Angeles) data center. Checking https://panel.dreamhost.com/index.cgi?tree=support.dc& will show you if your services are in that location. We are working on getting this resolved as soon as possible.

Damn, chill people. Yes, you can’t get email, but if you REALLY have to send email, just use a backup gmail account. Not the best solution, but a partial solution nonetheless.

Seems like all my stuff is down; websites and mail.
Any ETA on a fix?

Alright everyone, please STAY CALM!
I too have no email and the life of my business depends on it.
They are working diligently to resolve this issue..
..this issue which should NOT be occurring!

OMG OMG OMG OH MY GOD!!!
I’M LOSING SERIOUS MONEY!!!!
Can I bill you Dreamhost?!?

Of course, web mail (squirrel mail) doesn’t work either and just says I’m specifying an invalid user or wrong password which is WRONG.

I don’t need to send email, I need to receive it.

This is not good for my business…and consequently may not be good for your business. Hope the fix is quick.

Yarp, my mail is down as well. Right after my KS completed. Ouch. :((

It’s not possible to check https://panel.dreamhost.com/index.cgi?tree=support.dc& because that require my IP address to be authenticed, but the email authentication is not delivered. Guess that guarantees that my services are impacted.

AAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!

Fix it now

Yea, my site is down too… not just mail.

My email is down, too. :(

So I literally hung up with a prospective client who is waiting on an email from me… I’m going to lose the client… I don’t want to email this prospective client from gmail, as that will not be impressive. Thank you for keeping us apprised of the status and all that you do. In my experience with DreamHost, these problems are rare. The timing is bad now, but I guess there’s never a good time for things to go down. I really need to email this prospective client, though… Godspeed!

I was just thinking that it has been a long time since I’ve had any Dreamhost issues. Here’s to fixing the problem as soon as you can. I’m going to clean my studio. Do work!

@Timothy – may I suggest that you check out https://panel.dreamhost.com/index.cgi?tree=billing.secure& once you have access to the DreamHost panel again? It will let you avoid having to get your IP address authenticated for access.

We will post an update soon, thanks for your patience!

My email is Down!!!!!!!

Everyone CALM DOWN! YES it sucks, but DreamHost is one of the most reliable things in my life! Yes technology costs us money when it doesn’t work but that’s simply the price we pay. How many came here just because they really really needed someone to bitch at?

email issues almost every day, and now this!

My home listing and sales clients are trying to reach me and it is imperative that this get resolved immediately! I have transactions in process today!

What is wrong with you guys?

No back up power for your servers, no back up servers. I lose money every time an issue like this happens. It’s almost common.

What is wrong with you guys?

Are the incoming mail queued somewhere at your datacenter now or is the incoming servers also having this problem?

I thought we’d resolved these problem, what the hell. My whole office is down… The more this keeps happening, and the longer it lasts, the more beer we have to consume..

As per our latest status post, we are on our way to restore all services. The update also addresses the reason why the usual backup power wasn’t readily available. We appreciate your continued patience.

how long will this take to fix? No website, no $$.

More like NightmareHost…

Any predictions?????????

The amount of people complaining on what is essentially a budget service.

Yes, they should have redundencies (im sure they do) but it doesnt always work out, sometimes the backups fail too, sometimes the tertiary backups fail and so on…

If you are that worried, pay more, and/or have your OWN redundency in place.

ouch. why is it every time something goes down for a small group, i happen to be in that group? i’m guessing the rest of you are in the same boat.

Seriously? What do I pay for here? Your system is glitchy even when it’s up and running – this seems to be happening way too often. GET IT FIXED!

We pay Dreamhost to keep things these running for us.

Remember when there used to be a 99% uptime guarantee on the website? I guess they took it down a while ago. I still have a screenshot for what it is worth.

Dreamhost is down so much on a quarterly basis that I have migrated nearly everything we own off of their WordPress, FTP, Email, and Subversion infrastructures in the last 6 months. It has gotten so bad that I host my own monitoring service to tell me when Dreamhost is down, so I know when to contact support. I should send them a bill for my monitoring services.

I started having email problems since last week…Not such a DREAM(host) after all. Really ticked off.
My friend owns and operates a web-hosting, private server services and all that Cloud stuff in Los Angeles.
If you would like reliable Hosting go to XFERNET.NET.
Dreamhost, you may contact me if you would like me to introduce you to them.

@Chris Crug LOL! Guilty.

I am not in CA so why are we affected?

email still down :/

I’ve rarely ever have issues, and I’ve been with DH for around 5 years. VPS with lots of sites, Dreamobjects, etc. Such a good deal overall that I can’t complain for a little email downtime!

Mail and FTP both are not working for almost an hour now..!

We’re done with dreamhost

This is ridiculous.

This is absolutely unacceptable. I’m losing money every minute this server is down.

My sites are down, down, down… :-(
Too bad. I have a lot of work to do.
I though the days of no-backup-plan NightmareHost were over a couple of years ago.

yay back up here!!

Websites are coming back up. Thanks for the quick fix!

I didn’t notice until my husband asked me why his password was being rejected. First thing I did was check here. I’m sure it so be fixed soon.

despite this interruption DH has been awesome… i think some of you guys are thinking about baddreamhost.com thats different story…

I get that it’s not working but no amount of threats or harsh words is going to fix it.

That’s twice in a month for me… DH, you’re killing me.

E-mail down for over an hour – you say at 3:05 EST –

Update July 16th, 2014, 12:05 PM PDT: We are happy to report that power is being restored now, and our operations teams are working diligently to ensure that every service is up and running again as expected.

Not!

Going on an hour…..

What’s the deal?

my sites are down even though only the mail servers are in the la datacenter. all of my domains are hosted on dwalin in the east/virginia datacenter.

@Dawn…. Because the mail server that your account uses is in California.

I’ve already convinced my client to move all of their services this fall to another provider. We’ve spent years dealing with issues with these guys. While all hosting companies I’ve worked with have had issues in QoS, none come close to the regularity of Dreamhost. We’re gone this fall after we finish up two other big projects (not hosted here!!! ;-) and free up our time some because we have a lot to move.

andrea:
I would really be happy if you could answer this:
All the incoming mail that should have come while your service is down – are they queued somewhere at Dreamhost or is there a possibility they have been bounced for not arriving your datacenter?

My website is not in the LA server and it’s still down and can’t connect FTP!?

So many email/webmail problems over the last month.

So sad.

No mail access, once again DH Nightmare ..

Sites are all back up. Thanks, Dreamhost, for fixing everything so quickly.

3:57 eastern, Not up here.

Our website is back up….

UP =D

IMAP email is still down.

It’s not on!

All my mail is down!

All of my sites are up but still no email….

Webmail is not logging yet…

Good luck fixing this ASAP. I used to be a Network Engineer so I understand. :-)

All your info says I should not be effected, but I am. No emails?? website is fine tho????

PROMPT:

Unable to connect to your IMAP server. You may have exceeded the maximum number of connections to this server. If so, use the Advanced IMAP Server Settings dialog to reduce the number of cached connections.

HUH????

IMAP email is not working

We are still working on restoring services, and making sure everything is running as expected after this power outage. To reiterate, the confirmed cause of this service interruption was that a power panel that our setup connects to went down. Because this panel exists in between our machines and our provider’s generators and UPS system, emergency power was unavailable. We are working on correcting this in the future. Sorry again about the inconvenience!

3pm (CT) still down and counting … ouch!

Mail is FUBAR here too.

Everytime something like this happens, I tell myself we’re moving from Dreamhost to Media Temple or some place like that. But then I get complacent once the problem is resolved.

THIS NEEDS TO BE FIXED IMMEDIATELY!!!!!!!!!!!!!!!

UPS in contorted location. I recommend investing a full backup generator system.

All mail accounts down.

Are incoming emails being stored or bouncing back to senders?

OMG! Help! People are Dying!

Mail down! ETA on return? Losing business!

EMAIL working!!
Thank you guys
Good job!

OK It’s working!!!!!

1.05PM PST

Thanks for the frequent updates, Dreamhost folks.

Email working here

Still no working!

I can not believe you have no redundant power supply!
That is basic!
How much longer will it take?

July 16th, 2014 at 1:09 pmAbsoluteZero Says:

It’s a lot easier if the machines connect directly to a battery backup or UPS system. I’ve seen guys use those fused power strips in racks before and they always blow the fuse and take down 5-10 machines at a time. I try to run a battery back-up per machine myself in my installations. It’s cheaper than downtime and prevents a hard crash on a RAID.

It would be nice if there was a text or email (to my alternate mailboxes of course)notification service to let us know when service is back up and running.

Although I understand that it is hard to forecast when IT issues like this will be completely resolved. It would be nice to announce when we can expect your next update.

I know I personally would like to get an update every 1-2 hours even if it is only to say, “we are still working on it”.

Keeping me informed is next to keeping it running.

E-mail is not working, do I have to do something in particular? I really need to send some information to a client.

email is working here! 4:09 EST . Thanks for resolving this so quickly!

spoke too soon….

Whew!! End of the World crisis averted! Thanks Dreamhost. Everyone can go ahead and resume their life and death email businesses!!! That was a pretty tense hour.

working but INTERMITENT.
Anyway someone is working to resolve it

July 16th, 2014 at 1:12 pmNeeraj Oberoi Says:

would this have an effect in Montreal, Qc.

Well… my sites WERE back up. Seem to be down again now. Sigh….

It’s really not a huge deal, but it is frustrating.

@jasong wins best comment! Lets keep this in perspective folks…

Thanks for the quick resolve everyone here is happy. Less than an hour!

Fun fact: A 99-percent uptime guarantee means a service has to be up and running for 361.34757801 days of each year to stay within that SLA. That means everything can be down for 3.65242199 days of the year, which is 87.65812776 hours or 5,259.4876656 minutes, and you’re still getting what you pay for.

If you need more reliability, there’s a simple solution: pay more money, either by switching to a more expensive service provider or colocating your services yourself for increased redundancy (or do both). No matter who your host is, things will ALWAYS break and it will ALWAYS be at the worst possible time for somebody. (Case in point: Google’s Gmail going down for 41 minutes right before the last big holiday weekend. )

I share everyone’s frustration, but I also don’t expect Taco Bell to offer valet parking.

I’m back up.

again DOWN

What is the current situation?

I think some people are missing that important point that the downtime was NOT just for emails. Entire websites, custom servers, and services like MySQL and Subversion were down.

All my sites and emails are back up

I’m back up. Thank you DH for your work in this.

Website was back up for about 15 mins – NOW down again as of 1:15 PM PST.

Eveything is working here. A bit slow, but working.

again UP :)

@Billy Suratt

Bill wins!!!!

Email up, but still getting same email delays like we do every other day! Do something about this once and for all please.

E-mail is not working. Been real slow the last few weeks.
When it is wooking long delays.

As per our latest update: We were in the process of restoring all affected services when the same panel (see previous update) went down again. We are very sorry about this added inconvenience. Before the panel lost power for the second time, we began rerouting power to other power distribution units as a precaution. We are now in the process of moving services to those new circuits and will update this status post again as soon as we have more information.

Hey guys,

Can I just say “Thanks for the updates”? There’s nothing worse than a company that sticks its head in the sand when there’s a problem.

and down again…

No email accounts at all here in the UK.

Another fun fact: This isn’t the first time our services have been down this year at Dreamhost. I count 8 incidents in the last 2 months alone.

Yes, we have moved stuff that it more important to other hosting providers and pay slightly more for it.

Yet another fun fact: According to http://www.dreamhost.com/web-hosting/ “Our 100% uptime guarantee means we take uptime seriously”

and again DOWN :(

Up for a hot minute just after I sent an email telling everyone to send me email to Gmail. Then sent an email saying its back. Now it’s gone. You’re making me look like a crazy liar!

Well… my sites WERE back up. Seem to be down again now.
And it IS a big deal especially if you are in the middle of sending emails and it give you the error that it cannot connect to the server.

Email down. Multiple sites down. Don’t know what has happened to you guys. You USED to be most reliable, lately, its one thing after another. The “quirky” Newsletters you send out- may be starting to show that customers need to “read between the lines”. This isn’t a “game” or ‘fun”… you’re a business that other businesses rely on! Every update, just makes things worse! I think it’s time, for those of us who rely on you, we moved to a company who did not think everything is fixed with a “sorry for the inconvenience”.

Down – Up – Down. What is up? Not Dreamhost mail…

Anything new?

Just moved to DreamHost first week of this month. This is not what I would call a warm welcome.

No mail, no life, all is down

Since 2003, I have been under the impression that your servers had backups in Texas. When did this stop? For what I pay, this is not acceptable.

@jon, to answer your question, here is approximate timeline and how mail was handled during this incident:
12:20: disabling mail machines, so anything that came in before is queued and anything after is rejected
1pm: enabling mail machines again (save for homiemail-sub4), so some mail is again coming in and queueing
1:10pm (second power outage): disabling mail machines again, so no mail is coming in
1:35pm: enabling mail again, so we mail is starting to come in again, but homiemail-sub4 is taking a bit longer

Started working at 1.05pm PST. Now down again — incoming IMAP server down!!!

No email accounts at all here in Costa Rica.
Mail appears to be going up and down.

If you have business critical services on DreamHost, you’ve made a big mistake. Find a reliable host with some actual accountability if you need your services to work. Yeah DreamHost has a 100% uptime guarantee, but all that means it that for every hour they’re down, they have to credit you about $0.30 (on the rare occasions like this one where they actually admit there’s a problem). That’s not a lot of incentive for them to keep things running smoothly.

Am curious about what John says – the website does claim a 100% uptime guarantee, which is admittedly a ridiculous claim, but also one that isn’t being met. I’m not saying the world is ending, but if someone offers a guarantee, shouldn’t it mean something?

Hey suckers!, you answered my question before I asked it. Kudos to your psychic abilities. If only you could’ve predicted this outage.

@Billy Suratt is correct. All IT infrastructures will experience issues eventually. I came from Bluehost and when they had a failure, I’d contact them and they’d lie out their butts. Once the outage made the newspapers, they’d still try to BS me.

So be thankful for the honest information about what happened and frequent status updates. At Bluehost, they’d be ignoring me unless I sent them traces, email header information etc, all busy work knowing full well they were down. I jumped through their stupid hoops to find out they knew all along it was them on several occasions. They lost a customer not because of their outages, but because they were intentionally dishonest and wasted my time with contempt.

With Multiple data centers why isn’t their failover for all services between the different data centers… I thought that was a big point of having data centers in different locations…

As per our latest update: We have completed moving most of shared web servers off of the affected power panel. Once these machines boot back up our shared service will be 100% restored and we will be continuing work on restoring mail. As a reminder, this is only affecting a small number of shared web services (and soon none at all) and mail that’s located in our US-West (Los Angeles) data center. No other data centers or MySQL services were affected.

Folks:

The IMAP server is down and has been intermittently for days. I get incoming email, but can’t send anything. I run my business with this hosting service; why is it so inconsistent? (Not the first time IMAP has shut down, and yet, no refunds for downtime.)

Please address this. Researching other providers because this is ridiculous.

Not true Andrea – we are in Dallas and our mail is not working! Quietly freaking out in Dallas – NOT LA – unless LA handles our mail server…..

I host several websites and have clients who are really upset about this outage. Do we know yet how long before services will be restored? (Even if it’s a rough estimate)

This reminds me of the time when I was working for an ISP years ago where I was rolling up a V.35 cable after an equipment swap and the end of the cable swung down and hit precisely on the reset button of the news server. Since news was by far the busiest server, in terms of disk ops, it corrupted a significant portion of the file system. After several unsuccessful attempts at fsck’ing, I realized that a server rebuild was in order. What was supposed to be a simple outage and circuit cutover turned into an all night maintenance event. I think I left the building about the time the sun was coming up. I certainly don’t miss those middle of the night maintenance windows.

Reporting from Florida, we are down as well.

I was supposed to send a bunch of job search emails today!
C’est la vie!
I’ll just go to the park, lay down on the grass and watch the clouds.
:)))))))))) Life goes on

Thank you guys! Congratulations for the work!

Things happen folks. If you don’t like DH, go somewhere else like Billy said and pay more. Guaranteed you’ll spend more money for less quality. E-mails going down for a few hours every once in a while happens no matter who you host with. It’s just what happens when you rely on something that’s plugged into an outlet. The last company I worked for was a fairly large corporation, and had all of their own internal servers. The site as well as the emails for 1600 people went down every so often despite having an entire IT department. It’s the nature of things. Keep up the good work DH team! Thanks for jumping on this and keeping us all posted.

Site with dedicated IP not back up.

Another site is not displaying properly.

I’m going along with the messages that are telling people to get over themselves. If your business is so completely dependent on email being up 100% of the time, pay a lot more for a dedicated email server.

A.

WHOLE.

LOT.

MORE.

Seriously. Do you realize how little you pay for the amazing services you get?

Relax. It’s not healthy to get worked up like that.

Connie: it doesn’t matter where you are… I am in Atlanta, and my email is handled in the LA data center. Check the link in the next to bottom paragraph at top of page to see where you’re located.

Josh: Multiple data centers does not mean they’re mirroring servers/data across other data centers in the country… If that’s what you want, be ready to spend 100x more then Dreamhost.

p.s. MAJOR giggles from those of you who don’t understand what the location of a server means in relation to YOUR location.

Hint: it doesn’t mean squat.

@Connie: Sorry about the confusion. Your mail server location is showing at https://panel.dreamhost.com/index.cgi?tree=support.dc&

As per our latest update: All the affected shared web servers are now using a different power source and are fully operational. Our mail machines are coming back online one after each other. We are getting close to being fully restored and appreciated your continued patience in the meantime.

July 16th, 2014 at 2:08 pmDaniel Crooks Says:

@Nix Nix 100% for the clever kid in the class we are in Brisbane Australia….

OMFG NOOOOOOOOOOO I JUST GOT TRANSFERRED TO THIS SERVER A COUPLE WEEKS AGO AS AN “UPGRADE”! PUT ME BACK WHERE I WAS WARM AND HAPPY AND HAD NO DOWNTIME!!! THIS IS A MAJOR DISASTER, A CALAMITY OF UNPRECEDENTED PROPORTIONS, WITHOUT THIS ACCESS MY PRIVATES ARE SURE TO SHRIVEL UP AND FALL OFF!!! WHAT KIND OF P.O.S. PROVIDER DOES NOT HAVE REDUNDANCY FOR A POWER FAILURE, PARTICULARLY WHEN THEY ARE LOCATED IN EARTHQUAKE CENTRAL????? BUNCH OF PHUCKING MONKEYS!!!!! I FART IN YOUR GENERAL DIRECTION!!!!!!!

Can someone please pass the mashed potatoes? Thanks.

err.. gravy too please. Thanks. Carry on.

July 16th, 2014 at 2:17 pmDaniel Crooks Says:

Billy Surat “I share everyone’s frustration, but I also don’t expect Taco Bell to offer valet parking”

That analogy should be on a t shirt!! Awesome!!!

My sites are back! (Hopefully the email will all trickle in now too.)

Steve

Mail down all day; there are always workarounds but basically productivity is 50% down the tubes for today.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 10th 10pm PDT (4 hrs) RESOLVED
1:49 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 10th 10pm PDT (4 hrs) RESOLVED

Posted (July 8th, 2014 at 1:49 pm PST) by andrea

We will be conducting network maintenance in our US-West (Los Angeles) Data Center starting at 10pm PDT. This is to upgrade some of our current equipment to increase network redundancy and capacity. We estimate the maintenance window to be 4hrs, during which we will be moving groups of servers from the old network uplink switch to the new ones. Each of these moves may cause up to 30 second of service pause, and we will be very closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the MySQL (shared or VPS) machines are located. In order to check if you will be affected, please check the DreamHost panel for the location of your MySQL server. If you host dynamic web sites with us and your mysql server is at the Los Angeles location, those may also be affected indirectly, but there will be no impact on static sites or email service during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update July 11th, 2014, 1:09 AM PDT: We are happy to report that the maintenance, including the follow-up testing phase, has been successfully completed, with no noticeable impact on related services. Thank you again for your patience while we were working on it! We are closing the comments, but if you have any questions or concerns, please feel free to contact tech support, and we’ll be happy to help.

Update July 11th, 2014, 12:38 AM PDT: We finished moving all the related servers to the new uplink switches. What remains now is reviewing and monitoring to make sure the move was indeed successful. We’ll update this post again as soon as the maintenance is complete.

Update July 10th, 2014, 11:33 PM PDT: The maintenance is still going well without any noticeable impact, and we’re continuing to keep a close eye on the process. We appreciate your patience!

Update July 10th, 2014, 10:33 PM PDT: We are going through the steps to move from the old switch to the new ones, and everything is going as expected. We’ll update this post periodically throughout the process.

Update July 10th, 2014, 10 PM PDT: We are commencing the maintenance as scheduled. Comments are now open, so please let us know if you notice any issues with your service that may be related. As a reminder, this maintenance will only be conducted on a section of the data center where the MySQL (shared or VPS) machines are located. In order to check if you will be affected, please check the DreamHost panel for the location of your MySQL server. If you host dynamic web sites with us and your mysql server is at the Los Angeles location, those may also be affected indirectly, but there will be no impact on static sites or email service during this time.

This entry was posted on Tuesday, July 8th, 2014 at 1:49 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network infrastructure maintenance – June 19th 6PM PDT (2 hours)
12:17 pm

Network infrastructure maintenance – June 19th 6PM PDT (2 hours)

Posted (June 19th, 2014 at 12:17 pm PST) by Brandon

On Thursday June 19th, starting at 6:00PM PDT we will be replacing one of the network switches in our mail infrastructure. We expect no major impact to customers accessing mail. This is a proactive maintenance, and the previous maintenance in this series was executed seamlessly. We expect this maintenance to perform the same.

Update, 6:43 PM Pacific Daylight Time: And everything is complete! All went as planned. If you have any questions regarding this, please let us know by contacting Support.

Update, 6:37 PM Pacific Daylight Time: The maintenance is proceeding as planned without any customer-facing impact. Hopefully it is going so well no one even notices. If you have any questions, though, please always feel free to contact our support team.

Update, 5:58 PM Pacific Daylight Time:  Okay, we are starting!  As mentioned before, we do not anticipate that this will result in customers seeing any issues, but if you do have questions or concerns don’t hesitate to contact our support team.

This entry was posted on Thursday, June 19th, 2014 at 12:17 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Power issues in our US-West Data Center (Los Angeles) (RESOLVED)
10:57 pm

Power issues in our US-West Data Center (Los Angeles) (RESOLVED)

Posted (June 13th, 2014 at 10:57 pm PST) by Steven

We have experienced two power outages lasting 20 minutes or longer on one of our racks in the Los Angeles data center. There are twelve servers that have been impacted by this:

hovhaness, kaelin, mozart, hindemith, honegger, nielsen, milhaud, nystroem, holmboe, koechlin, kashif, and karlowicz.

We are actively investigating the cause of the outages so we can ensure they do not continue to occur.

This entry was posted on Friday, June 13th, 2014 at 10:57 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Outage (RESOLVED)
9:57 am

DreamObjects Outage (RESOLVED)

Posted (June 13th, 2014 at 9:57 am PST) by Dan J

Our cloud engineering team are currently investigating 503 errors being reported by the DreamObjects service. At the moment it appears some servers are unresponsive and that is causing the errors. There is not yet an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support contact support for assistance.

Update June 14th, 2014, 10:20 AM PDT: Stability on the DreamObjects cluster has been restored. Requests appear to be resolving properly now that the system has had time to re-balance itself. Our test are reporting properly now. If you do have any questions or concerns, please contact support contact support.

Update June 14th, 2014, 3:45 AM PDT: Unfortunately, there isn’t much new to report. Test runs are hitting their timeouts instead of failing. Requests are, by and large, going through. We should have more information in an update later today.

Update June 13th, 2014, 7:50 PM PDT: We’re still monitoring DreamObjects to ensure the upgrade went successfully. Our testing software is still showing some errors on some passes. We want to be sure our tests complete without error, so please bear with us for a bit longer. We will continue to update with more info as this progresses.

Update June 13th, 2014, 5:06 PM PDT: Right now, we’re still monitoring DreamObjects to ensure the upgrade went successfully. Our testing software is still showing some errors, so we’re still not quite ready to call this resolved, however, all informal tests we perform show service as largely restored. We will continue to update with more info as this progresses.

Update June 13th, 2014, 3:56 PM PDT: Our cloud team has confirmed that the update has completed. We are now testing, and seeing positive results on some clusters, but not all tests are working 100% yet. We’re hopeful we’ll be able to restore full access relatively soon and we’ll update as soon as we have more info! Thank you for your patience and understanding while we work to resolve this issue.

Update June 13th, 2014, 2:27 PM PDT: Our cloud team has confirmed that the physical process of updating the software is near completion. We don’t have a firm ETA as to when services will return to normal, but we’ll update this post as soon as we have any further updates for you. Thank you for your patience and understanding while we work to resolve this issue.

Update June 13th, 2014, 1:25 PM PDT: Our cloud team continues to apply updated software to the DreamObject cluster. Hopes remain high that once fully updated this will correct the stability issues we’re seeing. We shall continue to give updates here as we make progress.

Update June 13th, 2014, 12:40 PM PDT: So far the DreamObjects servers appear to be accepting the upgraded ceph version properly. Our engineers continue to apply those updates as quickly as possible to stabilize the cluster.

Update June 13th, 2014, 12:10 PM PDT: The new ceph version is currently being rolled out to the DreamObjects machines. The upgrade itself appears to be going smoothly and as quick as possible.

Update June 13th, 2014, 11:25 AM PDT: Our cloud team currently is performing an emergency software upgrade to address the issue. We believe some of the problems are resolved in a newer version of ceph. As previously noted all customer data should remain safe.

Update June 13th, 2014, 11:05 AM PDT: The DreamObjects service continues to be offline while our cloud engineers work to stabilize the cluster.  All data stored is however safe at this time.

Update June 13th, 2014, 10:40 AM PDT: Several of the machines are still not responding as expected, our cloud team continues to work to correct this issue.

Update June 13th, 2014, 10:20 AM PDT: Our team is still working to recover some unresponsive machines. There isn’t an ETA yet but we have our full cloud team on the issue.

This entry was posted on Friday, June 13th, 2014 at 9:57 am and is filed under DHO, General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses to “DreamObjects Outage (RESOLVED)”

Again? :(
Can you tell us what the problem is this time? So at least we’ll know if DreamObjects is a dependable service, or if we should treat it as “beta”.
:(

Hi,

what about your post that aborted the upgrade of Ceph. It was not possible then and now your solution for this mess is to upgrade Ceph?

Update May 28th, 2014, 2:37 PM PDT: During testing of the DreamObjects upgrade, we found a bug that forced us to abandon the upgrade and continue at a later date. Once the issue is resolved, we will schedule another upgrade window. We apologize for this, and thank you for your patience!”

@sadface we understand this is frustrating, we are doing everything we can to ensure DreamObjects is as rock solid as possible.

@Me we did abort an earlier attempt to upgrade to a new version, for several reasons, including the potential of exceeding our commitment on our maintenance window, and we identified another issue that we needed to resolve before upgrading.

That issue has now been resolved, and we had planned on doing the upgrade during a maintenance window at a later time, however, we feel that the upgrade will fix the problems we’re seeing now, so are moving forward with the upgrade to accomplish this.

Mine is still not working!!!!

@Trevor, can you send me a link to what you’re testing?

Will there be compensation for those of us who where unable to get work done due to the outage?

Still unable to access my buckets…

About 4 PM all objects I checked were available. Now
503 errors for many objects at 10:45
Example:
https://objects.dreamhost.com/argentum/ARU/ogac/chm47/thumbnails/AST_7588.jpg

Mine still not working, i’m getting internal server errors!! :((((
It’s been more than 12h……

12:30 AM Things seem to be working OK.

June 14th, 2014 at 3:21 amAdrian Chapman Says:

Still intermittent here in UK at 10:00 GMT.

My buckets were briefly available earlier, but now I can’t access anything again. This is pretty bad.

3:20 AM 503 Service Unavailable

4:15 AM 503 Service Unavailable

FYI
04:33:39 PDT (though I’m in Poland, 13:33)

e.g.
http://static.jajajanenene.com/coldFeetTeaserOct2012.mp4

503 Service Unavailable
No server is available to handle this request

—————————————-

After being off yesterday, 13:00-ish PDT , it did play again at about midnight PDT

honestly, i’ve prepared for the worst case scenario – all data gone…

10:00 AM
Every object checked so far is OK.
uploading and downloading are as fast as ever.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WebFTP Upgrade – June 10th 11pm PDT (1 hours) (RESOLVED)
3:59 pm

WebFTP Upgrade – June 10th 11pm PDT (1 hours) (RESOLVED)

Posted (June 6th, 2014 at 3:59 pm PST) by Brian@dreamhost

On Tuesday June 10th, starting at 11:00pm PDT we will be upgrading our WebFTP software. WebFTP (https://webftp.dreamhost.com) will be temporarily unavailable as we make this change. The duration of this maintenance is 1 hour, from 11pm PDT till 12am PDT June 11ith. If you have any questions about the upcoming maintenance, please contact our support team.

Update June 10th, 2014, 11:45 PM PDT: The upgrade has been successfully completed! There should be no issues as a result. Be sure, however, to contact us if you see any strange behavior in the WebFTP software. Thanks!

Update June 10th, 2014, 11:00 PM PDT: The WebFTP service will be offline shortly to upgrade the software. As noted, we only expect this to take about one hour. Please contact support if you have any questions or concerns.

This entry was posted on Friday, June 6th, 2014 at 3:59 pm and is filed under Announcement, FTP Outages, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Outage (RESOLVED)
12:30 am

DreamObjects Outage (RESOLVED)

Posted (June 6th, 2014 at 12:30 am PST) by chalise

We are currently experiencing an outage with our DreamObjects service. We will update with more information as it becomes available.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate contact support contact support for assistance.

Update June 6th, 2014, 8:33 AM PDT:  OK DreamObjects is online. Our automated tests are passing and the cluster is in its expected state.  If you do find anything that isn’t functioning as expected then please don’t hesitate contact support contact support for assistance.

Update June 6th, 2014, 6:45 AM PDT: Work continues. We are unable to provide an ETA as yet but do hope to provide more details by the next post.

Update June 6th, 2014, 5:10 AM PDT: We are still working to resolve the downtime. It is, however, taking longer than hoped due to high load on some machines. Once again, we thank you for your patience.

Update June 6th, 2014, 3:10 AM PDT: We believe it may take another couple of hours to resolve the downtime. The pace is picking up as we move along, however, so it may be sooner.

Update June 6th, 2014, 2:45 AM PDT: Unfortunately, nothing new to report at this time.

Update June 6th, 2014, 2:00 AM PDT: We are making progress but are still working to resolve the downtime. We apologize for the trouble and thank you for your understanding.

Update June 6th, 2014, 1:00 AM PDT: We are still working to resolve the outage. No additional details are available at this time.

This entry was posted on Friday, June 6th, 2014 at 12:30 am and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “DreamObjects Outage (RESOLVED)”

I’m not getting any emails from any of my accounts hosted through Dreamhost. When will this be fixed?

June 6th, 2014 at 2:15 amVincent Duke Says:

ETA?? Anything? This is part of my core services I use in my online businesses need this up otherwise i lose clients and money.

Client will launch new website in few hours. She needs to update some contents and she cant access her web. update pls!

LoLLoLLoLLoLLoLLoL

June 6th, 2014 at 8:36 amVincent Duke Says:

So since your saying no other information is available does that mean you don’t know where the problem is? Or what it is? That would at least give me an idea if you can provide some details

DreamObjects is now functioning again. Our automated tests are passing and the cluster is in its expected state. I apologize for the delay.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Time for PHP Upgrades!
4:50 pm

Time for PHP Upgrades!

Posted (June 5th, 2014 at 4:50 pm PST) by mir

Starting June 24th, we will be upgrading domains on shared hosting running PHP 5.2* to our recommended version of PHP 5.4.

Fear not! If you need a little more time to upgrade, you can delay until November by going to the Dashboard of the DreamHost Panel and selecting “see what sites need upgrades” and by pressing “delay”. For full information about the upgrade, and how to delay (and even how to tell if you need to delay), please check out this upgrading PHP article in our wiki.

*PHP 5.2 has been completely unsupported by PHP.net since January 2011, so if you were planning on delaying, we strongly recommend that you begin testing right away.

As always, if you have any questions or concerns don’t hesitate to contact our support team.

This entry was posted on Thursday, June 5th, 2014 at 4:50 pm and is filed under Announcement, DreamHost Features, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

5 Responses to “Time for PHP Upgrades!”

Ok I have to admit this is somewhat disappointing. Anyone with a clue has been on 5.4+ for a long, long time now.

The title of your post felt like you’d be offering PHP 5.5. I mean 5.6 is almost out the gates, why do you folks lag so badly behind all the time?

Our email has not been working since yesterday at the point you started doing the PHP upgrades. Can someone please check on that and get back to us? It’s an emergency.

My email has been out as well. Not cool.

Both our affected sites have been out, or partially out, since Friday morning! I’ve tried to get someone’s attention via message to support team but nobody’s responding.

My email has been out for the past two days as well. I sent them a message through an alternate email. I can’t even get into my webpanel because it wants me to validate my IP through email.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance Affecting US-East (Virginia) June 5th 9pm – June 6th 1am PST(4 hours) (RESOLVED)
11:53 pm

Network Maintenance Affecting US-East (Virginia) June 5th 9pm – June 6th 1am PST(4 hours) (RESOLVED)

Posted (June 3rd, 2014 at 11:53 pm PST) by Brian@dreamhost

On Thursday, June 5th, starting at 9pm PST we will be performing network maintenance on one of our data centers:

US-East (Virginia)

This maintenance will last 4 hours, and improve communication between our data centers and the internet at large.

The changes will require propagation across the internet via the BGP routing protocol. As a result, your service may appear to be down during the maintenance for a short period that should not exceed 1 minute. Service interruption will largely depend on your ISP and how quickly the changes register within their network. To help identify any problems, we will have our staff monitoring the changes from multiple locations and ISPs.

We will provide updates as we move through the maintenance.

To see which data center your site is hosted in, please look at Status->Data Centers section in the panel.

Comments will be opened when the maintenance starts so that we can assist you with any issues you may be noticing.

If you have any questions about the upcoming maintenance, please contact our support team.

Update June 5th, 2014, 10:09 PM PDT: And just like that, we’re done! We thank you for your patience! If you’re seeing any issues, please contact our support team right away!

Update June 5th, 2014, 9:58 PM PDT: Everything is going as expected as we’re seeing very little downtime. We are moving through each subnet of the network slowly but surely!

Update June 5th, 2014, 9:28 PM PDT: We’re still methodically adding and removing routes, and everything continues to go smoothly. Some customers may see very brief interruption of service, however, we’re monitoring to make sure those interruptions stay brief.

Update June 5th, 2014, 9:05 PM PDT: Jumping right in, we’re making small changes to routes that can cause some brief moments of downtime. Each change is updated, verified to be working, and committed.

Update June 5th, 2014, 9:00 PM PDT: We’re here and setting up. Comments are open now, as well, so if you have an issue, post that here.

This entry was posted on Tuesday, June 3rd, 2014 at 11:53 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 30th
8:40 am

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 30th

Posted (May 27th, 2014 at 8:40 am PST) by Justin@DH

On Monday, May 12th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

banshee biggs bulger burke castiglia colossus coonan cummiskey dagger dalitz deadpool dio fay fitzpatrick floyd galactus greengoblin havok hawkeye hoffa joker jubilee kakuji karpis mclean metropolis mothman nightwing pistone provenzano spiderman spillane storm walsh watanabe wolverine zwerbach

As well as these web VPS host machines:

homie-vserver02 homie-vserver06 homie-vserver47 homie-vserver48 homie-vserver55 homie-vserver56

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.

This entry was posted on Tuesday, May 27th, 2014 at 8:40 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – May 28th 10AM PDT – 12PM PDT (2 hrs) (EXTENDED) (RESOLVED)
10:17 am

DreamObjects Maintenance – May 28th 10AM PDT – 12PM PDT (2 hrs) (EXTENDED) (RESOLVED)

Posted (May 22nd, 2014 at 10:17 am PST) by Justin@DH

As part of our continued efforts to improve our cloud services, on Wednesday May 28th from 10am-12pm PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this two hour maintenance window, we will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes improvements for large object support when using the OpenStack Swift API.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate contact support contact support for assistance.

Update May 28th, 2014, 2:37 PM PDT: During testing of the DreamObjects upgrade, we found a bug that forced us to abandon the upgrade and continue at a later date. Once the issue is resolved, we will schedule another upgrade window. We apologize for this, and thank you for your patience!”

Update May 28th, 2014, 12:55 PM PDT: We’re extending this maintenance window another 2 hours from the original estimate. We should be done at 2pm PDT. We apologize for any inconvenience this causes.

This entry was posted on Thursday, May 22nd, 2014 at 10:17 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 12th
1:13 pm

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 12th

Posted (May 9th, 2014 at 1:13 pm PST) by Justin@DH

On Monday, May 12th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

aquaman barker batman bonanno brasco bruno buchalter bugsy cable capone castellano catwoman cloak cohen cyclops dhaccountbackup eastman escobar gambino gambit genovese giancana gotti gravano greene hulk iceman kingpin lansky legs lepke magneto mangano masseria mcgurn moran nitti obelix penguin phantom riddler robin rothstein scalise schultz siegel spilotro sugeknight terranova thing trafficante venom

As well as these web VPS host machines:

homie-vserver01 homie-vserver04 homie-vserver05 homie-vserver07 homie-vserver08 homie-vserver09 homie-vserver10 homie-vserver11 homie-vserver12 homie-vserver13 homie-vserver139 homie-vserver14 homie-vserver15 homie-vserver152 homie-vserver16 homie-vserver17 homie-vserver177 homie-vserver18 homie-vserver19 homie-vserver197 homie-vserver20 homie-vserver21 homie-vserver22 homie-vserver23 homie-vserver230 homie-vserver24 homie-vserver25 homie-vserver26 homie-vserver27 homie-vserver28 homie-vserver29 homie-vserver30 homie-vserver31 homie-vserver32 homie-vserver33 homie-vserver34 homie-vserver35 homie-vserver37 homie-vserver396 homie-vserver42 homie-vserver44 homie-vserver45 homie-vserver46 homie-vserver53

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.

This entry was posted on Friday, May 9th, 2014 at 1:13 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

14 Responses to “Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 12th”

The correct link to the “data centers” page is https://panel.dreamhost.com/index.cgi?tree=support.dc

Yes. Some great news. Earthzine.org has been down for most of the week, ‘hosted’ by you.

I love the upgrade but I get
#2013 – Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110
I cant work, please check it!!!

site is down!

Web panel is down! Hard to log in to check anything if you can’t even get to panel page to look up your webserver. Another Dreamhost failure!

Panel.Dreamhost.Com is inaccessible at the moment, looking for resolution soon.

All my sites have dropped and I am hoping to 8 hours any reply from support. Very disappointed!

Getting calls from clients that email isn’t working.

Hi Dreamhost
Our website and email are down. They have not been working for a few days. Is there something I need to do get things happening again? Help!
M

Great, I just moved my sites to you and within days they are down. So much for the 100% guaranteed up time huh? So what exactly is 100%? I have moved servers before and NEVER did it take more than an hour or two depending on the size and location. Also I never got an email warning of this so I was racking my brain trying to figure out what I may have done. Been waiting in chat for another 30 minutes.

My site is down for a few days as well. Technical support is not answering, even if i sent a few emails. Sad

My website is also down, and won’t respond at all since I sent in a request for a reboot. I’m submitting a second request now, but I’m doubting it’ll work.

All client sites are experiencing incredibly slow load times and/or time outs when accessing their WordPress dashboards.

I could not import a small 500k SQL file into a MySQL database without getting a Bad Gateway error.

What is going on???

Hi DH,

I’m having problems as well, my two website are not running, I have written two mail to support but I haven’t gotten any answer. We deserve to know what is going on, and how to procede with our customers.

Please send any kind of communication.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS Maintenance in US-West (Los Angeles) – May 14th 10PM PDT (2 hrs) (RESOLVED)
8:04 am

Web VPS Maintenance in US-West (Los Angeles) – May 14th 10PM PDT (2 hrs) (RESOLVED)

Posted (May 7th, 2014 at 8:04 am PST) by Justin@DH

On Tuesday, May 13th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver01 homie-vserver04 homie-vserver05 homie-vserver07 homie-vserver09 homie-vserver10 homie-vserver11 homie-vserver12 homie-vserver13 homie-vserver14 homie-vserver15 homie-vserver16 homie-vserver17 homie-vserver18 homie-vserver19 homie-vserver20 homie-vserver22 homie-vserver23 homie-vserver25 homie-vserver26 homie-vserver28 homie-vserver29 homie-vserver30 homie-vserver31 homie-vserver32 homie-vserver33 homie-vserver34 homie-vserver35 homie-vserver42 homie-vserver44 homie-vserver45 homie-vserver46 homie-vserver53 homie-vserver197 homie-vserver396

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update May 14th, 2014, 10:57 PM PDT: Tonight’s maintenance is complete!

Update May 14th, 2014, 10:30 PM PDT: We apologize for the short notice as we are going to proceed with this maintenance tonight. We are getting under way in just a few minutes!

Update May 13th, 2014, 10:39 PM PDT: We are postponing this for another day (to be determined).

This entry was posted on Wednesday, May 7th, 2014 at 8:04 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Web VPS Maintenance in US-West (Los Angeles) – May 14th 10PM PDT (2 hrs) (RESOLVED)”

I’m a little confused by this post. How has this not happened yes but there are updates from April 29th?

@Phil: Our bad — someone copied the information from a similar maintenance on April 29th and left the progress reports in. Thanks for the heads up and sorry for the confusion!

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Discussion list scheduled maintenance May 7th 6pm PDT (4hrs) (RESOLVED)
4:51 pm

Discussion list scheduled maintenance May 7th 6pm PDT (4hrs) (RESOLVED)

Posted (May 6th, 2014 at 4:51 pm PST) by andrea

On Wednesday, May 7th, starting at 6pm, we will be performing maintenance on all our discussion list services. We will be upgrading the the Mailman software that’s running those lists from 2.1.14 to 2.1.17. Along with this upgrade we’re also turning on DKIM capabilities. This is a very important change for those of you who have been affected by Yahoo’s DMARC compliance change.

During this maintenance window your list administration interface will be unavailable. Messages posted to your list during this time will be queued and delivered later when the maintenance has been completed.

We apologize for the short notice, and we appreciate your understanding and patience while we perform this upgrade. We will open comments during the maintenance window and will be closely monitoring the progress. This update only affects discussion lists, and no other services (such as regular email, web hosting, or mysql hosting) are affected.

If you have any questions or concerns, please don’t hesitate to contact support, and we’ll be happy to help.

Update May 7th, 2014, 10:15 PM PDT:  We are happy to report that the discussion list maintenance has been completed successfully, and your discussion lists are now using Mailman version 2.1.17. This upgraded version of Mailman has the benefit of offering two options to make a discussion list message correctly pass a receiving mail server’s checks. For more information on those options and how to set them in the Mailman admin panel, please see DreamHost Wiki: Discussion Lists.

Thanks again for your patience while we were taking care of this. Please note, your list administration interface should be accessible again and your queued messages are in the process of getting delivered and should catch up shortly. If you have any questions or concerns regarding this, please contact support and let us know.

Update May 7th, 2014, 9:55 PM PDT:  We have completed the upgrade on all our discussion list services, and our techs confirmed that everything has gone as planned. We are now going into the testing and monitoring phase for the rest of the maintenance window, but at this time, your list administration interface should be working again and all queued messages should start being delivered.  If you see any issues with either of those please contact support and let us know so we can help you.

Update May 7th, 2014, 8:53 PM PDT: The upgrade is nearly complete. We will update this post again in an hour, or less if there is a status change.  Please contact support if you have any questions regarding this maintenance, and thank you for your patience during this process.

Update May 7th, 2014, 7:51 PM PDT:  The upgrade is proceeding smoothly as expected. We will update you periodically, once an hour, or sooner if there is a status change. As a reminder, you may experience limited access to your list administration interface and messages that are posted to the list are queued to be delivered later. We appreciate your continued patience during this maintenance.  If you have any questions, please contact support and we will happily help you.

Update May 7th, 2014, 7:14 PM PDT: Two machines are already done! We are going to the next, and will update this post again as soon as its upgrade is complete. We appreciate your patience while we’re working towards completing the maintenance.

Update May 7th, 2014, 6:17 PM PDT: We have completed work on one of the servers for tonight’s maintenance, and all is well! We are proceeding to the second server. Just a reminder, any messages sent to the discussion lists throughout the duration of this maintenance will be queued for later delivery, and the discussion list admin interface will be unavailable.

Update May 7th, 2014, 6:01 PM PDT:  We are starting work on the first server in our maintenance. During this time, any messages sent to the discussion list will be queued for delivery and the discussion list admin interface will be unavailable. We appreciate your patience during this process.

This entry was posted on Tuesday, May 6th, 2014 at 4:51 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Discussion list scheduled maintenance May 7th 6pm PDT (4hrs) (RESOLVED)”

Why are you upgrading to one version before current?

@micha – We already had this maintenance planned before 2.1.18-1 was released on 06-May-2014. Our prep work was with the 2.1.17 version, so we went ahead with that.

This is good news!

Note that if you are the owner of an existing list, you’ll still have to set the from_is_list option yourself in your admin web panel. It’s on the General Options page.

Default is exactly as it is now (No). Either set Mung From or Wrap Message. Searching the web on those terms will get you info on which you might want to use.

For Mung From to work, the MTA that DH is using for Mailman has to be configured to DKIM sign Mailman messages. No idea whether they’ve done that yet. If a DH admin sees this maybe he/she can answer.

I just tested my list with Mung From set. It may just be that Yahoo is extra slow tonight, but I am not yet seeing messages from Yahoo arrive at my Yahoo test account. They are arriving at other (non-DMARC) servers and the archive.

In any case, thanks to DH for moving ahead with this upgrade!

Hi David! I’m an admin here and had a hand in the Malman upgrades tonight.

The ‘Mung From’ option replaces the email’s original sender information with that of the discussion list. DreamHost does not add DKIM keys to the discussion list domains (ex, list.mydomain.com) so the discussion list server does no DKIM signing of mail sent from the list. With no DKIM signature, it is not possible for the DKIM verification to fail, resolving many DMARC and DKIM related bounces. Mailman’s interface does ‘advise’ that the discussion list domain have a DKIM key and sign emails, but this is not required for the ‘Mung From’ setting to function.

If you’re using the ‘Mung From’ option and mail sent from a DKIM-signing domain is not being received after being re-sent out from the discussion list, please contact support! They’ll be happy to help investigate, and can check the discussion list servers and logs to see precisely what is occurring.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Thursday, May 8th, 12am to Friday, May 9th, 12am PDT
4:33 pm

Live Chat Unavailable Thursday, May 8th, 12am to Friday, May 9th, 12am PDT

Posted (May 6th, 2014 at 4:33 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Thursday, May 8th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 12am PDT the next morning, May 9th. Thank you for your patience and understanding.

This entry was posted on Tuesday, May 6th, 2014 at 4:33 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects connectivity issues (RESOLVED)
7:02 pm

DreamObjects connectivity issues (RESOLVED)

Posted (May 4th, 2014 at 7:02 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case, and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Sunday May 4th 7:40pm PDT: We are happy to report that have found a resolution and all related services are functioning as expected. Please accept our apologies for any inconvenience this has caused. As always, if you have any questions or concerns, please contact support with the details, so we can assist you.

Update Sunday May 4th 7:22pm PDT: Connectivity to DreamObjects services has been mostly restored, so you should be able to access your objects/upload/download. We are continuing to monitor the service status, and aren’t resolving this incident yet. We apologize for any inconvenience this may have caused.

This entry was posted on Sunday, May 4th, 2014 at 7:02 pm and is filed under DHO, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost panel connectivity issues (RESOLVED)
5:55 pm

DreamHost panel connectivity issues (RESOLVED)

Posted (May 4th, 2014 at 5:55 pm PST) by Seohee

We are currently experiencing connectivity issues with the DreamHost panel. We are looking into the cause now and working towards restoring functionality.

We appreciate your patience and are sorry for the inconvenience. We will provide you with an update shortly.

Update Sunday May 4th 6:40pm PDT: The issues plaguing the DreamHost panel have been resolved and all is operating again as expected. We are resolving this status post yet will continue working to find the root cause of today’s outage. As always, if you have any questions or concerns, please don’t hesitate to contact our technical support team, and we’ll be happy to help.

Update Sunday May 4th 6:12pm PDT: We have restored functionality to the panel, but we are continuing to investigate the root cause of the incident. We will update again as soon as we know more or if there is status change. We appreciate your continued patience as we work towards the final resolution.

This entry was posted on Sunday, May 4th, 2014 at 5:55 pm and is filed under Control Panel Outages, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “DreamHost panel connectivity issues (RESOLVED)”

May 4th, 2014 at 6:01 pmrhetoricmonkey Says:

Thank you.

Accessing junk mail via webmail doesn’t seem to be working either.

Dreamobjects is also not accessible, was very slow earlier as well.

Justin Wells, sorry about that. We’re looking into that right now. We will start a new status post if this is indeed a wide-spread issue.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects connectivity issues (RESOLVED)
8:34 pm

DreamObjects connectivity issues (RESOLVED)

Posted (May 3rd, 2014 at 8:34 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case, and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Saturday May 3rd 11:15pm PDT: We are happy to report that have found a resolution and all related services are functioning as expected. Please accept our apologies for any inconvenience this has caused. As always, if you have any questions or concerns, please contact support with the details, so we can assist you.

Update Saturday May 3rd 9:55pm PDT: Our investigation continues, and we have isolated the issue to a specific instance. We appreciate your continued patience while we’re working towards resolution.

Update Saturday May 3rd 9:25pm PDT: We have noticed some residual issues and continue to investigate the root cause. Apologies for any inconvenience this may be causing.

Update Saturday May 3rd 9pm PDT: It appears that all related services are operational again. We’re still looking into what caused the disruption, and will update with details as soon as we have them.

This entry was posted on Saturday, May 3rd, 2014 at 8:34 pm and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – Software May 5th 10am PDT (8 hrs) (RESOLVED)
3:12 pm

DreamObjects Maintenance – Software May 5th 10am PDT (8 hrs) (RESOLVED)

Posted (May 2nd, 2014 at 3:12 pm PST) by andrea

On Monday, May 5th between 10am and 6pm PDT, our Systems Engineering team will continue the maintenance that started on May 1st to upgrade Ceph, the underlying storage technology powering DreamObjects. This upgrade includes general performance improvements as well as additional features for managing the cluster.

During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. However, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Monday May 6th 10:58am PDT: We are happy to report that the above maintenance is has successfully completed yesterday. We will be posting here again when the next maintenance is coming.

Update Monday May 5th 1:20pm PDT: All tests have completed and things are looking good after our RAM upgrades. We are about half way through our maintenance and we are expecting it to go smoothly from here on out. If there are any further issues we will update this status post. Otherwise, we will update it when we have completed the maintenance. Please don’t hesitate to contact us if you notice any further issues with your DreamObjects service.

Update Monday May 5th 12:45pm PDT: We ran in to a couple of small issues that caused connectivity problems but have been able to quickly address them. We’ve upgraded RAM on a couple of key components and will be performing a couple of tests shortly, so that should help keep us in the clear for the rest of the maintenance period. At this time there shouldn’t be any customer facing issues, but we will update this post again shortly with more details.

Update Monday May 5th 12:22pm PDT: This maintenance has now begun. As noted above, there may be some brief connection problems, but we will try to keep any downtime to a minimum. We will update this post again when the maintenance is complete, or if there is any other noteworthy news in the meantime.

This entry was posted on Friday, May 2nd, 2014 at 3:12 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – Software May 1st 6PM PDT (10+ hrs)(POSTPONED)
3:36 pm

DreamObjects Maintenance – Software May 1st 6PM PDT (10+ hrs)(POSTPONED)

Posted (April 25th, 2014 at 3:36 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Thursday May 1st from 6pm-12am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this six hour maintenance window, we will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Friday May 2nd 3pm PDT: As noted in an earlier update, we will have to schedule another maintenance window to finish what we couldn’t complete last night. Sorry about any inconvenience this may have caused. Please see the new DreamObjects maintenance post for more details.

Update Friday May 2nd 4:43am PDT: The trouble we ran into is preventing us from completing the maintenance tonight. So, we will need to schedule another time to complete it. Everything is in a safe state right now so we do not anticipate any issues. Of course, please report any problems to support (link above)! Thanks!

Update Friday May 2nd 4:02am PDT: This is taking much longer than originally anticipated but we are continuing our work to get it completed as soon as possible. We do not have a new estimate yet on time to completion but will keep you updated.

Update Friday May 2nd 1:58am PDT: Unfortunately, we ran into a little trouble and need to extend this by another two (2) hours. Again, we greatly appreciate your patience and understanding.

Update Friday May 2nd 12:59am PDT: The original window for this maintenance was six (6) hours. Unfortunately, we do need to extend this to eight (8) hours. Additional updates will be provided as they become available. We appreciate your patience!

This entry was posted on Friday, April 25th, 2014 at 3:36 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – Hardware April 30th 6PM PDT (6 hrs) (RESOLVED)
3:34 pm

DreamObjects Maintenance – Hardware April 30th 6PM PDT (6 hrs) (RESOLVED)

Posted (April 25th, 2014 at 3:34 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Wednesday April 30th from 6pm-12am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this six hour maintenance window, we will be upgrading hardware on the machines that oversee the storage nodes.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Thursday May 1st 11:05am PDT: This is just a quick follow up to let you know that last night’s maintenance was successful and well within the announce maintenance window. Please note, there is another DreamObjects maintenance tonight. You can find the details under the other DreamObjects post on this status page. As always, if you have any questions or concerns, please don’t hesitate to contact support for assistance.

This entry was posted on Friday, April 25th, 2014 at 3:34 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects connection troubles (RESOLVED)
9:39 pm

DreamObjects connection troubles (RESOLVED)

Posted (April 24th, 2014 at 9:39 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Thursday April 25th 12:47am PDT: This issue should now be resolved. We are able to verify that all instances are now working. If you are still seeing problems, please contact support for assistance. We thank you for your patience during this issue!

Update Thursday April 25th 12:20am PDT: We are now making some progress bringing DreamObjects back online. Some Objects should appear up, however, they may be slow, or showing some errors. We are continuing to investigate these issues and appreciate your patience.

Update Thursday April 24th 11:34pm PDT: We are continuing to look through logs and perform other steps to investigate the problem with the load balancer. We appreciate your patience while we work towards resolution.

Update Thursday April 24th 10:45pm PDT: Unfortunately, no news yet; we are still investigating the root cause. Just to repeat, this is only affecting our DreamObjects service.

Update Thursday April 24th 10:20pm PDT: We are continuing to investigate the root cause for the load balancer issue. We appreciate your patience and apologize again for the inconvenience this may be causing.

Update Thursday April 24th 10pm PDT: The signs point to a load balancer issue that’s causing inability to handle any incoming connections. We are continuing to investigate and will update you again as soon as possible.

This entry was posted on Thursday, April 24th, 2014 at 9:39 pm and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “DreamObjects connection troubles (RESOLVED)”

Good luck. Not sure if it’s related, but I’m seeing errors when trying to connect to a mySQL database on rollingpin from my web server clinch.

@Scott — I am so sorry for the problems! rollingpin isn’t related to this one, but we are working on that one as well, so you should see improvement very shortly!

Noticed missing DO at exactly 9PM
At midnight Do was still down but 5 minutes later it seems fixed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS Maintenance in US-West (Los Angeles) – April 29th 10PM PDT (2 hrs) (RESOLVED)
8:22 am

Web VPS Maintenance in US-West (Los Angeles) – April 29th 10PM PDT (2 hrs) (RESOLVED)

Posted (April 22nd, 2014 at 8:22 am PST) by Justin@DH

On Tuesday, April 29th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver73
homie-vserver86
homie-vserver87
homie-vserver88
homie-vserver91
homie-vserver92
homie-vserver93
homie-vserver94
homie-vserver95
homie-vserver97
homie-vserver100
homie-vserver233

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update Thursday April 29th 10:28pm PDT: And just as the party starts it gets shut down! That’s how you know it’s a real good party… Alas, this maintenance has been completed and no VPS hosts were harmed in the process.

Update Thursday April 29th 10:07pm PDT: We have begun! Downtime should be short and limited to the reboots for each VPS host machine. Stay tuned for updates!

Update Thursday April 29th 10:05pm PDT: We are about to get under way!

This entry was posted on Tuesday, April 22nd, 2014 at 8:22 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning April 28th
5:24 am

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning April 28th

Posted (April 22nd, 2014 at 5:24 am PST) by Justin@DH

On Monday, April 28th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

alouettes argonauts baku bears beijing bills bluejays brasilia bridgetown browns brussels bujumbura canberra cannons cardinals cavs chivas crew crusaders d-fenders eagles earthquakes eskimos fever fire force grizzlies herediano hurricanes indians islanders jazz jets kabul lightning lizards luanda minsk nassau nuggets oilers olympiacos ottawa ouagadougou outlaws penguins pigwidgeon pirates pistons pride rapids rockets santiago shock sofia stjohns thimphu thrashers tigercats vienna vikings waratahs yankees yaounde

As well as these web VPS host machines:

homie-vserver73 homie-vserver86 homie-vserver87 homie-vserver88 homie-vserver91 homie-vserver92 homie-vserver93 homie-vserver94 homie-vserver95 homie-vserver96 homie-vserver97 homie-vserver98 homie-vserver99 homie-vserver100 homie-vserver233

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.

This entry was posted on Tuesday, April 22nd, 2014 at 5:24 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

5 Responses to “Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning April 28th”

Will this fix email block issues on these servers?

Outside DNS new IP what? Don’t we always use ns1.dreamhost.com etc so that IP is irrelevant?

What tez said. How are IPs relevant if we use ns1.dreamhost.com and ns2.dreamhost.com?

IPs are relevant if you’re using DH for hosting but not using ns1/2.dreamhost.com for DNS. If you’re using DH for DNS, no worries.

i got a few emails about this, which had 0 details about what was being moved. i contact support and they had no clue about this, asking to have that email fwd’d to them… does one hand know what other is doing around here. ffs

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost Web Panel Changes April 22nd, 2 PM PDT
2:44 pm

DreamHost Web Panel Changes April 22nd, 2 PM PDT

Posted (April 18th, 2014 at 2:44 pm PST) by mir

We are excited to share some impending changes to your DreamHost Panel (https://panel.dreamhost.com). Next Tuesday we’ll be making a few minor organizational changes that mark the first preparations for some improvements to how you manage your DreamHost services. Based on suggestions and usage data we’re re-ordering and consolidating the navigation, moving a few items and removing some cruft. Please head here for full details (this document would be a good one to bookmark to look for any future alterations as well):

http://wiki.dreamhost.com/Panel_UI_Changelog

As always, if you have any questions or concerns, please don’t hesitate to contact support for assistance.

Thank you for choosing DreamHost!

This entry was posted on Friday, April 18th, 2014 at 2:44 pm and is filed under Announcement, DreamHost Features. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

47 Responses to “DreamHost Web Panel Changes April 22nd, 2 PM PDT”

Are the current downtimes due to these changes?

Same here, my website is down. Any news?

My sites are down as well and chat operators are busy. :(

I’m down too, but only on a sub domain website.

2 of our 3 domains are down – we’re basically out of business until service is restored.

I got response on chat:
“The changes to the panel are not related to your web hosting services. turns out that the account has processes are being killed by our procwatch daemon. Procwatch is a daemon that runs constantly on shared servers to monitor the usage of RAM/CPU and execution time so that no single user can use an inappropriately high percentage of the shared resources and impact the overall health of the server or the server’s ability to serve all users’ pages.”

I am quite sure that is not true, I disabled robots, and there are no any scripts, that could cause this.

My website is down also. Cannot do business without it. Hope it gets resolved quickly. Cannot get a hold of chat operator.

Got a hold of someone. There was a looping piece of code that was consuming a lot of CPU. Seems to happen every so often on my site. Hopefully you can all get someone and mention the same thing.

Well, this was the straw that finally got me to sign up for a web site uptime monitoring service. Time is money and if Dreamhost can’t keep my site up, I might have to move to someone who can.

What’s the deal? Can you guys not make updates without downtimes??

All our email addresses are down. Is this related?

Our email addresses are down

Our email accounts are not working.

Some (not all) email addresses have been returning the following message for 10 minutes or so:
Temporary authentication failure.

My site has gone down as well. Any confirmation that this is related or not?

It’s 3:42 eastern time, my website has been down since early this morning. I was first notified by one of my users that my site was down at 8:52 am. I can access my two subdomains dev.pvda.org and scores.pvda.org, but not my main site http://www.pvda.org.

Why does DreamHost have so much down time? My user’s are starting to complain. My site needs to be up so that people can submit entries to horse shows that close for entries at certain times. People get really mad when they can’t enter a horse show because my site is down.

I am also having a devil of a time using the new Pydio webftp service. This morning I used it for the first, and I saw files listed in my home directory that I had not created!!! One was a directory called “alexshamy.com” and one was a file called “Annexes 2011-2012.pdf”. Those are gone now, but I can’t access my wp-content/pvda/wp-uploads folder (equivalent to the “wp-content/uploads” folder).

It sounds like crazy things are happening. I thought my account had been hacked when I saw those extra files. I entered two trouble tickets, so hopefully someone will look at it.

Please bring the system up soon!!!

Our email accounts are not working either. This is getting old – real old.

What is going on Dreamhost? Get it together…

My Outlook account is totally useless today….and I have a mountain of work that won’t be completed and a Statewide meeting in just three weeks. Thanks a lot for doing these “upgrades” and/or “consolidations” during normal business hours!!!!!!!!

Mine’s down, too. It’s only a basic WordPress build with some theme modding, so I don’t see how it could be using a vast amount of resources or the like. I also don’t see that looping code could be it either as I’m not altering the basic code of the WordPress engine! Really not happy that we weren’t notified of this being a possibility during the upgrade process.

My site is also down. It was up this morning but now I am getting the dreaded 503 error. Please help! Google Adsense is in the process of reviewing the site and this may have ruined my chances this month.

My websites also went down because my users were wiped-out from my VPS account. Files were still there as I could SFTP in but going to Web Panel>Users>Manage Users showed zero users. I had to get into chat and they re-enabled my missing user accounts but my sites are still showing, “error id: “bad_httpd_conf”

Anyone else have this today? Checking my Account Status seems to indicate my webserver may have also been changed.

Getting 503 again..had this problem in the morning and it was resolved.. it just a basic wordpress site,don’t understand what resources it is using for being down like this.So frustrating

My site is down, too, and today were are about to launch a new product :-(

Websites down for over eight hours now. No response back from two tickets submitted three and eight hours ago.

Fortunately, my site is accessible again.

my site is down

My site is down…when this will be up..Any one please respond..its been a day my site is down…

It seems to be working now. But what about the emails I didn’t receive yesterday?

My site has been down 23 hrs! and now is really slow. Is anybody else having this issue?

Again???????? Crickey… get your shit together Dreamhost

Half the time I load this Blog I get an older version from February … with no posts beyond that date.

ALL of my sites are completely down because they can’t connect to MySQL. The following error is on all of my sites:

Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110

I have opened a ticket (#6284149), but based on these posts here and the fact that no information of an outage is on this site, I’m a bit worried.

All of my websites are down too! :(

I’m still not able to login to the control panel. I get the following error message (of course tried several browsers with cookies/cache cleared):

OOPS! Something just went terribly wrong:

Can’t call method “err” on an undefined value

Please contact DreamHost Support for details.

My site is also down or very slow. Sorry to say Dreamhost but this will be the last time!
I need a reliable webhost!

Really frustrating especially if your money is made that way.

My site can’t be opened for 24 hours.
The worst webhost I have seen so far.

Googlebot can’t access your site.
Over the last 24 hours, Googlebot encountered 27 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 78.5%.

For me it’s finaly stable. My guess is that there were apache problems on my server, you can read about them in forum. It’s misfortunate that support doesn’t admit that, you litteraly have to prove them they are wrong. I lost 6h trying to find where is problem, and prove that it’s up to them by deleting all files from server and uploading single html page.

Annnnnd again we are down. great.

From support, a mass email explaining my now-2-day downtime:
—————————

We’re very sorry that you’ve been affected by this issue. Our systems engineers have been investigating and we feel that we’ve gotten closer to resolution, but we’re not completely there yet. The problem is that certain apache web services are running out of virtual memory which in turn is causing 60 second delays followed by timeouts on a number of customer sites.

We’ve provided the engineers with the details on your account, so that they can work toward a permanent resolution and will be restarting the the apache instance on which your domains run to bring your domain back online – either already or very soon.

Unfortunately, there have been some repeat cases for some of the apache web service instances, so you may experience downtime again, for which we are very sorry. We are monitoring these services and will restart them again should this occur.

We are keeping your tickets open until permanent resolution has been reached and will update you periodically.

Please accept our apologies, due to the high influx of support you are receiving a mass response. If you had a specific question that was not answered, please reply and let us know. If you submitted a callback request, it has been credited back to your available pool.

Please let us know if you have any other questions or concerns, we’ll be glad to help!

I have a WordPress site hosted by Dreamhost that has been down over 24 hours. When will mindfulnesscoach.com.au be back? Mindfully Yours – John

Ugg… this is bs. I wrote tech support and put in a ticket over 24 hours ago and didnt even get a ‘we received your problem – here’s your case #’ reply.

Website down and no explaination from dreamhost . what is the point of your status website? the no news just scares customers to new providers .

The email server is down. What’s wrong?

The panel is very very slow today…

I am getting a 403 Forbidden error when trying to access the following page:

http://www.nehrlinggardens.org/www/administrator/index.php

Forbidden

You don’t have permission to access /www/administrator/index.php on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Without access to this page i can not log into joomla or civicdm. We need to be able to access this info. Please help

-Cindy

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WebFTP Upgrade – April 18th 10pm PDT (4 hours) (RESOLVED)
11:52 pm

WebFTP Upgrade – April 18th 10pm PDT (4 hours) (RESOLVED)

Posted (April 17th, 2014 at 11:52 pm PST) by Oscar

On Friday April 18th, starting at 10:00pm PDT we will be replacing our WebFTP software. AjaXplorer is no longer being developed and has been replaced by Pydio! During this maintenance, we will be removing our AjaXplorer WebFTP client and installing Pydio. WebFTP (https://webftp.dreamhost.com) will be temporarily unavailable as we make this change. The duration of this maintenance is 4 hours, from 10pm PDT till 2am PDT April 19th. All WebFTP references will remain on the DreamHost panel and will link to the new software afterwards. If you have any questions or concerns, please don’t hesitate to contact support for assistance.

UPDATE: April 19, 2014, 12:40 AM PDT: I apologize for the delay on this post. The good news, however, is that we are done! The upgrade is complete and no issues were found during testing. If you do experience any problems with the WebFTP interface please contact our Technical Support team. Thanks!

UPDATE: April 19, 2014, 12:00 AM PDT: Nothing new to update at the moment. We are still working to complete the upgrade for WebFTP.

UPDATE: April 18, 2014, 11:30 PM PDT: We are still working on it. Stay tuned for further updates!

UPDATE: April 18, 2014, 11:02 PM PDT: The maintenance is moving along. We have hit one snag (webftp.dreamhost.com is loading with a blank page) but do not anticipate going outside of the quoted 4 hour window. Hang in there!

UPDATE: April 18, 2014, 10:37 PM PDT: We are a go! We are beginning work on the upgrade now and will post updates frequently (every 30 mins or less). WebFTP is currently offline.

UPDATE: April 18, 2014, 10:21 PM PDT: Just a heads up that we are currently getting prepared to move forward with the upgrade!

This entry was posted on Thursday, April 17th, 2014 at 11:52 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance April 23rd 6PM PDT – April 24th 4AM PDT (10 hrs) (RESOLVED)
2:40 pm

DreamObjects Maintenance April 23rd 6PM PDT – April 24th 4AM PDT (10 hrs) (RESOLVED)

Posted (April 16th, 2014 at 2:40 pm PST) by Justin@DH

As part of our continued efforts to improve our cloud services, on Wednesday April 23rd from 6pm-4am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. We will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Thursday April 24th 2:20pm PDT: Just a quick update to let you know that the maintenance went down without a hitch. If you have any questions or concerns about your DreamObjects service, please don’t hesitate to contact support for assistance.

This entry was posted on Wednesday, April 16th, 2014 at 2:40 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.9 Major Release Now Available
11:55 am

WordPress 3.9 Major Release Now Available

Posted (April 16th, 2014 at 11:55 am PST) by Mika E.

It’s been a rough few weeks in the WordPress world, but WordPress 3.9 has finally been released.

All the security features of 3.8.3, plus some amazing new editor features are now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

For all the details, check out the official WordPress release notes for 3.9 “Smith”. We’re most excited about the editor changes. You can drag and drop media right into the text editor, but also you can just copy and paste from Word right into WordPress. No more remembering to click on that ‘paste from Word’ icon no one can find!

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If you run into issues after your upgrade, remember to report any problems you have with specific plugins and themes to those developers on WordPress.org! Every single plugin developer was contracted, prior to release, to remind them to test, so we’re hoping that worked. It was a very large email list.

This entry was posted on Wednesday, April 16th, 2014 at 11:55 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance, US-West (Irvine) Data Center Thursday April 17th 10pm PDT (RESOLVED)
3:55 pm

Network Improvement Series Maintenance, US-West (Irvine) Data Center Thursday April 17th 10pm PDT (RESOLVED)

Posted (April 15th, 2014 at 3:55 pm PST) by Ralph

On Thursday April 17th, starting at 10pm PDT, we will be conducting network maintenance in our US-West (Irvine) Data Center. This is the final maintenance in our network refresh series, Network Improvement Series. The expected duration of this maintenance is 4 hours. During this maintenance, we will be upgrading the software on our core router. This is a tested process that we have done before with no noticeable customer impact. We anticipate similar results this time, with the slight possibility of brief downtime as we go through each rows of machines while moving links to and from the backup switches.

Our order of operations is as follows:

1. Moving links, one by one, from the main core router to our temporary switches. ( Can cause less than a minute of service downtime for sites, mail and other services, as links associated with those services are moved.)

2. Reboot the core. ( No impact.)

3. Moving the links back to the successfully rebooted core. ( Can cause less than a minute of service downtime for sites, mail and other services, as links associated with those services are moved.)

We are excited that this maintenance will conclude our efforts to refresh our network. We have already seen improvements of our network stability as we have progressed through our Network Improvement Series of Maintenances.

Comments will be open during the maintenance so that we can assist you in real time, and verify if your issues are related to this maintenance.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.
If you have any questions or concerns please contact the DreamHost Support Team.

Update Friday April 18th 1:10am PDT: We are now resolving the maintenance, along with our Network Improvement Series. Tonight’s maintenance completed without any major interruption of service, and we saw no customer impact. Thank you for your patience during this update, as well as during the Network Improvement Series, which has helped us solidify our network for the future.

Update Friday April 18th 12:45am PDT: All links have been moved successfully back to the main core router. There was no noticeable service impact throughout this maintenance, other than a quick issue with DreamObjects availability. Our network team is performing some clean up prior to calling this maintenance resolved. We are not expecting any further issues from this maintenance.

Update Friday April 18th 12:30am PDT: The upgrade and reboot has completed successfully. We are now in the process of moving the links back to the main core router. During this time we could see some service interruption lasting less than 1 minute as each link is moved.

Update Thursday April 17th 11:45pm PDT: All of the links have moved over successfully with no reported service interruptions. We will now be moving on to the reboot of the core router so that we can upgrade the software. We do not expect any service interruption for this step of the maintenance.

Update Thursday April 17th 11:10pm PDT: DreamObjects is back in working order. We are now going to move the links. This will cause less than a minute of service interruption as we move the links from our core router to the backup routers. We will move quickly through this process as we see that links are properly working after each move.

Update Thursday April 17th 11:00pm PDT: We are currently investigating a possible issue with DreamObjects. This may or may not be related to this maintenance. We want to make sure that this issue is resolved prior to continuing.

Update Thursday April 17th 10:30pm PDT: Step 1 was completed successfully. Step 2 is also not expected to be service impacting. Once those two steps are complete, we will move on to actually moving the links from the core, to our backups. During that time there will be less than a minute of service interruption as we move each link.

Update Thursday April 17th 10:10pm PDT: During these first couple of steps we will be setting up links on our backup routers so that we can move the redundant links. There is no expected service impact during this stage of the maintenance. We will update this post once these steps are completed successfully.

Update Thursday April 17th 10:00pm PDT: Our team is onsite and ready to begin work on this maintenance. As we progress through each step we will update here and respond to any comments or questions you may have.

This entry was posted on Tuesday, April 15th, 2014 at 3:55 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS Maintenance in US-West (Los Angeles) – April 15th 10PM PDT (2 hrs) (RESOLVED)
8:55 am

Web VPS Maintenance in US-West (Los Angeles) – April 15th 10PM PDT (2 hrs) (RESOLVED)

Posted (April 10th, 2014 at 8:55 am PST) by Justin@DH

On Tuesday, April 15th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver71
homie-vserver74
homie-vserver75
homie-vserver76
homie-vserver78
homie-vserver79
homie-vserver81
homie-vserver82
homie-vserver83
homie-vserver84
homie-vserver89
homie-vserver196

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

UPDATE: April 15, 2014, 23:16 PDT: All done! Everything went as planned.

UPDATE: April 15, 2014, 22:50 PDT: This is progressing as expected so far with no issues. We will post another update in 30 minutes, or sooner if there is any change.

UPDATE: April 15, 2014, 22:11 PDT: We have started the maintenance work outlined above. As noted, we do not anticipate downtime for the listed host machines to last very long.

This entry was posted on Thursday, April 10th, 2014 at 8:55 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost and Heartbleed – Notes on OpenSSL Vulnerability
3:22 pm

DreamHost and Heartbleed – Notes on OpenSSL Vulnerability

Posted (April 9th, 2014 at 3:22 pm PST) by Brett

This vulnerability only affects domains which use secure hosting / SSL.
– If you purchased a secure certificate from us, you’re safe! There is nothing you need to do.
– If you generated keys with us but purchased the certificate from another provider, you will have to check with them to see if you need new ones. If you do, you will have to get the new certificate through them and then we’ll be happy to provide you with new keys!
– If you’ve purchased a certificate from us and are now using it with another provider, you will need to ensure that other provider was not affected by this vulnerability. If they were, our technical support team will be happy to issue you a new certificate and keys!

As soon as we learned of the “Heartbleed” OpenSSL vulnerability, we began to patch any and all systems that it may have affected. Fortunately this was a very small subset of our systems and was mostly isolated to a small group of mail machines. As of early yesterday, all of our systems are patched. As a preventative measure, we are also re-keying the certificates on any systems with that bug. We have no reason to believe that any of those machines have been compromised, but in the interest of proactive security, we feel that changing SSL certificates is the best option.

DreamHost.com was not vulnerable, but the machines that redirected traffic to our actual site were. This was corrected quickly and those machines will also have their certificates re-keyed.

We can confidently say that our shared servers, VPS guests, and dedicated machines are NOT vulnerable to this issue because they run Debian “Lenny” and/or “Squeeze”. The most common version of OpenSSL on our network is 0.9.8o-4squeeze14, and the “HeartBleed” vulnerability in OpenSSL’s heartbeat module exists in versions 1.0.1 and 1.0.2-beta.

If you have any questions or concerns, please don’t hesitate to contact our support team.

UPDATE: April 10, 2014, 11:08 PM PDT: Please note, that earlier today we re-issued our email certificates. To resolve any exception or error when it shows up regarding mismatch, you can try accepting the certificate. If that doesn’t help, you will need to remove your email address from your email client then re-add it and accept the security certificate/exceptions and server identities during that process.

This entry was posted on Wednesday, April 9th, 2014 at 3:22 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Network Maintenance US-West (Irvine) Data Center Thursday April 10th 10 PM PDT (4hrs) (RESOLVED)
12:14 pm

Scheduled Network Maintenance US-West (Irvine) Data Center Thursday April 10th 10 PM PDT (4hrs) (RESOLVED)

Posted (April 8th, 2014 at 12:14 pm PST) by andrea

On Thursday April 10th, starting at 10pm PDT, we will be performing network maintenance in our US-West (Irvine) Data Center. During the scheduled 4-hour window, we will be migrating the network layer from some older hardware to the newly designed setup. This is expected to be a near impact-free maintenance; all services will continue to work throughout the migration. However, you may experience momentary disruption when you have a service running on the specific portion of the network being moved. This includes anyone having any services in that data center as well as DreamObjects customers. As with any maintenance, we will follow a pre-tested process and will proceed slowly to ensure that we are able to address any issues that may occur.

Please note, we had to renumber a few apache web service IP addresses due to this maintenance. If you’re using our name servers, then this change is seamless for you; you don’t have to do anything. However, if you’re using outside name servers for your domain, you will have to update your A/Host record with your other provider if your current record set up with them matches any of the below:
208.113.144.3 changed to 208.113.145.57
208.113.148.2 changed to 208.113.148.193
64.90.60.2 changed to 64.90.60.89
67.205.12.3 changed to 67.205.13.22

Comments will be open so that we are aware if you’re seeing anything out of the ordinary. If you have any questions or concerns pre-maintenance, please contact support with the details, and we’ll be happy to help. You can check if your services are in the above data center in the DreamHost Panel.

UPDATE: April 11, 2014, 3:50 AM PDT: We are happy to announce maintenance has completed! Thanks once again for your patience at the extended maintenance window!

UPDATE: April 11, 2014, 3:30 AM PDT: Maintenance is still ongoing, however, everything continues to move along smoothly as expected, and we are getting close to wrapping things up. Thanks for your patience, and we will update again once the maintenance has completed.

UPDATE: April 11, 2014, 2:08 AM PDT: It looks like we’re running a bit past schedule, but everything has been and is continuing to move along as expected. We’ll update this post again as soon as the maintenance is complete (likely within an hour from now) or if there is a status change). We appreciate your patience.

UPDATE: April 11, 2014, 1:08 AM PDT: We have been going through the customer-facing steps of the migration and there is some more left to do. No impact on the service, and we’re continuing to monitor to make sure it remains so.

UPDATE: April 11, 2014, 12:11 AM PDT: The maintenance is proceeding towards the public-facing steps. We are continuing to keep a close eye, and we’ll update this post within an hour or if there is any status changes.

UPDATE: April 10, 2014, 11:08 PM PDT: Just a quick update to let you know that we’re moving forward so far as expected. We’ll update this post again in a bit.

UPDATE: April 10, 2014, 10:08 PM PDT: We are starting the network maintenance as scheduled. We’ll be posting periodic updates as the migration proceeds and if there is any change in status.

This entry was posted on Tuesday, April 8th, 2014 at 12:14 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Server downtime related to security updates (US-West: Los Angeles and Irvine, US-East) – RESOLVED
7:23 pm

Server downtime related to security updates (US-West: Los Angeles and Irvine, US-East) – RESOLVED

Posted (April 1st, 2014 at 7:23 pm PST) by Elizabeth

We are performing unplanned maintenance on multiple servers that may result in a very brief amount of downtime. A recent security update was released that caused a problem. This error did not compromise security but it made operations on a random number of servers across three data centers fail.

We are correcting the error and restarting the services of the affected machines. We do not currently have an estimated time this will be completed, other than as soon as possible.

We sincerely apologize for the unplanned nature of this issue.  If you have any questions please contact support.

UPDATE: April 1, 2014, 7:40 PDT:  All services have been restored, and we do not anticipate any further issues. If you do see any problem with your hosting, please contact support and ask us to check it for you.

This entry was posted on Tuesday, April 1st, 2014 at 7:23 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail Server Sending Issues Due to RBL Block
12:03 pm

Mail Server Sending Issues Due to RBL Block

Posted (March 31st, 2014 at 12:03 pm PST) by Justin@DH

We are experiencing issues with customers ability to send mail to some mail providers, due to certain IP addresses belonging to our mail servers being listed on a RBL block. This can result in emails not being sent from your mail client, or emails being returned to their sender. Our abuse team is working with the entity blocking these ips, to get the issues resolved and the blocks removed as soon as possible.

We sincerely apologize to those customers who are being impacted by this issue and we will update this post when we have further details or news for you.

UPDATE: April 12, 2014, 10:50 PM PDT: We are happy to report that the Spamcop block situation has been under control for a little while now. We are continuing to keep an eye on anything that may cause the block again, and have recently shut down the major spammers. Even though we’re already very strict against outgoing spam, we’re also considering several system and policy enhancements to ensure continued unblocked status – not just with Spamcop but with other providers.

We are also looking into smaller blocks as they come up, so if you feel you’re affected, please provide us with a sample bounce error (dated the day you send it, not old ones, which may already have been resolved) with the full bounce email and headers of the original. Having this information is necessary to trace which IP or host is being blocked.

UPDATE: April 4, 2014, 1:00 PM PDT: As of last night Spamcop unblocked our three main email load balancer IPs, which should resolve the bulk of the problem for now. We are staying in contact with them to try and prevent the blocks from happening again while we make internal improvements to our network. We are doing our best to prevent further incidents going forward of course and are staying vigilant and alert to react rapidly to such an event.

If you receive additional blocked messages please provide us with a sample bounce error (dated the day you send it) with the full bounce email and headers of the original. Having this information is necessary to trace which IP is being blocked in case it is different from the three above that we are most acutely aware of.

We thank you for your patience with us in dealing with this disruption.

UPDATE: April 3, 2014, 12:45 PDT:  During the night, a couple more of our IP’s were added to the RBL block, causing further issues with mail being delivered from our system. We’ve addressed the issue on our end and have requested that those IP’s be de-listed. Mail going through these affected IP’s should begin to send again as DNS propagates. We are making progress on clearing up these blocks and thank you for your continued patience.

UPDATE: April 2, 2014, 9:02 PDT: We have identified the major contributor of the RBL block. The affected IPs are scheduled to be unblocked during the course of the day, but it’s going to take a while for everything to be cleared up due to DNS propagation. We appreciate your continued patience while we’re finalizing resolution of this issue. We are also actively looking into preventing this situation from happening again in the future.

This entry was posted on Monday, March 31st, 2014 at 12:03 pm and is filed under Email Issue, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS Maintenance in US-West (Los Angeles) – April 1st 10PM PDT (2 hrs) (RESOLVED)
7:17 am

Web VPS Maintenance in US-West (Los Angeles) – April 1st 10PM PDT (2 hrs) (RESOLVED)

Posted (March 30th, 2014 at 7:17 am PST) by Justin@DH

On Tuesday, April 1st from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver36
homie-vserver38
homie-vserver39
homie-vserver40
homie-vserver41
homie-vserver49
homie-vserver50
homie-vserver51
homie-vserver52
homie-vserver54
homie-vserver189
homie-vserver394

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

UPDATE: April 1, 2014, 22:57 PDT:  What? That’s it? Yes! Tonight’s maintenance has been completed. Please contact us (link above) if you find any issues with your services.

UPDATE: April 1, 2014, 22:15 PDT:  The maintenance has begun!

UPDATE: April 1, 2014, 22:05 PDT:  Just an update to let you know we’re just about ready to get underway with this! We will post another update when the maintenance begins.

This entry was posted on Sunday, March 30th, 2014 at 7:17 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Web VPS Maintenance in US-West (Los Angeles) – April 1st 10PM PDT (2 hrs) (RESOLVED)”

can’t check the data centers page, because Panel seems to be down. Panel timed out on me, and now I can’t log in – the process just seems to hang.

Panel is down?

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Maintenance for homiemail-sub1 (homiemail-master) email sub-cluster March 27th 7pm PDT (3hrs) (RESOLVED)
11:20 am

Scheduled Maintenance for homiemail-sub1 (homiemail-master) email sub-cluster March 27th 7pm PDT (3hrs) (RESOLVED)

Posted (March 24th, 2014 at 11:20 am PST) by andrea

On Thursday, March 27th from 7PM-10PM PST, we will be performing maintenance on our “homiemail-sub1″ (homiemail-master) mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. This new solution has undergone rigorous testing and has been successful on three sub-clusters already, all of them larger than this one, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion. This is the fourth time we’re doing this type of maintenance since January, so based on that experience, we were able to reduce disruption to the minimum.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster. It should state ‘homiemail-master’.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update March 27, 2014 8:08pm PST: This maintenance is now done! We are seeing no issues at this time and are resolving this maintenance. If you’re still seeing any problems with email, please let us know by contacting support.

Update March 27, 2014 7:55pm PST: At this time, we’re monitoring some changes we’ve made, ensuring that these updates will not cause problems.

Update March 27, 2014 7:20pm PST: We are now ready to make adjustments to our load balancers. This can cause some very brief moments of failed connections. If you see problems like this, reconnecting should help to resolve those issues. If we have any expected longer outages, we’ll update here.

Update March 27, 2014 6:53pm PST: We’re here and getting ready to start. We’ll update as soon as we’re ready to start the maintenance.

This entry was posted on Monday, March 24th, 2014 at 11:20 am and is filed under Email Issue, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » NS2 Hardware Upgrade – old IP address removed
5:31 pm

NS2 Hardware Upgrade – old IP address removed

Posted (March 18th, 2014 at 5:31 pm PST) by andrea

This is the follow up announcement we promised in our NS2.dreamhost.com Hardware Upgrade – IP Change Required for Vanity Name Servers post.

We are confident that this change will have only minimal effect because of the monitored and observed very low level of traffic through the old IP address (the change happened almost two months ago), but if you need assistance, please contact support with the details.

This entry was posted on Tuesday, March 18th, 2014 at 5:31 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS Maintenance in US-West (Los Angeles) – March 18th 10PM PDT (2 hrs) (RESOLVED)
9:43 am

Web VPS Maintenance in US-West (Los Angeles) – March 18th 10PM PDT (2 hrs) (RESOLVED)

Posted (March 17th, 2014 at 9:43 am PST) by Justin@DH

On Tuesday, March 18th from 10PM to 12PM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver57
homie-vserver58
homie-vserver59
homie-vserver60
homie-vserver61
homie-vserver62
homie-vserver64
homie-vserver65
homie-vserver66
homie-vserver67
homie-vserver69
homie-vserver292

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update March 18, 2014 10:09pm PST:We are beginning the maintenance now! As noted above, there will be a short period of downtime when the VPS host machines are rebooted.

Update March 18, 2014 10:41pm PST:And like that, it’s done! If you see any issues with your service please contact support via the admin panel. Thanks!

This entry was posted on Monday, March 17th, 2014 at 9:43 am and is filed under Data Center Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance March 18th 10AM PST (8 hrs) (RESOLVED)
3:21 pm

DreamObjects Maintenance March 18th 10AM PST (8 hrs) (RESOLVED)

Posted (March 11th, 2014 at 3:21 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Tuesday March 18th from 10am-6pm PST, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. We will be upgrading to Ceph, the underlying storage technology that powers DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

This entry was posted on Tuesday, March 11th, 2014 at 3:21 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Maintenance for homiemail-sub3 email sub-cluster March 13th 7pm PST (4 hrs) (RESOLVED)
4:19 pm

Scheduled Maintenance for homiemail-sub3 email sub-cluster March 13th 7pm PST (4 hrs) (RESOLVED)

Posted (March 10th, 2014 at 4:19 pm PST) by andrea

On Thursday, Mach 13th from 7PM-11pm PST, we will be performing maintenance on our “homiemail-sub3″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. During this time, we will also be making backend DNS changes to our mail system. While this shouldn’t cause any customer-impacting issues, we do acknowledge that possibility and will keep a very close eye at all steps during the maintenance. This new solution has undergone rigorous testing and has proven to be effective already on the “homiemail-sub5″ and “homiemail-sub4″ subclusters, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

Please note: Once you have determined that you are on homiemail-sub3 cluster, and if you are using outside DNS (ie. you are not the default DreamHost nameservers), you have to update the IP address for your ‘mail’ A record to the new one. The new IP address is 69.163.253.7 .

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update March 13, 2014 9:25pm PST: We consider this maintenance complete and successful. If you need any help with your mail service or anything else, please Contact Support with your details.

Update March 13, 2014 8pm PST:  At this time, the DNS change is complete. If you are using outside DNS (ie. you are not the default DreamHost nameservers), you have to update the IP address for your ‘mail’ A record to 69.163.253.7 .

We’re monitoring the changes and making sure everything works as expected. We’ll update this post again with any new information.

Update March 13, 2014 7pm PST: We are proceeding with the maintenance as planned. We will post periodic updates as we progress.

This entry was posted on Monday, March 10th, 2014 at 4:19 pm and is filed under Announcement, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Power issues in our US-East Datacenter (Virginia) (RESOLVED)
9:40 am

Power issues in our US-East Datacenter (Virginia) (RESOLVED)

Posted (March 10th, 2014 at 9:40 am PST) by Dan J

We had a power fluctuation in our US-East Datacenter around 9am PST today. This caused a large number of machines to reboot. We’re currently running on backup generators due to maintenance earlier this morning in this datacenter. All power has been restored now and appears to be stable, but we’re watching to make sure no other issues pop up.

Update March 10, 2014 10:30 am PST: It appears service has returned to normal now. We will continue to monitor the situation to make there are no lingering issues.  Thanks again for your patience. If you find that your site is not currently functioning as expected then please Contact Support.

Update March 10, 2014 10:24 am PST: Almost all machines are now online and functioning. We’re spot checking the last few unresolved problems.  Thanks again for your patience!

Update March 10, 2014 9:54 am PST: Most affected servers appear to have come back online correctly. Some servers however booted into an incorrect run level, which caused their hosted services to not start up properly. Our administrators are currently correcting those issues. Thank you for your patience!

This entry was posted on Monday, March 10th, 2014 at 9:40 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

46 Responses to “Power issues in our US-East Datacenter (Virginia) (RESOLVED)”

Sites are down on king-george for us…,.

Strange. Reboots normally come back up within a minute or two. This was more like 40 mins+.

Also, what was the “maintenance” earlier today that has caused failover to backup power? Hadn’t seen anything about this on @dhstatus or the status website!

Thanks for the update but why did it take 40 minutes to post something?

March 10th, 2014 at 9:47 amlucas vasconcelos Says:

my websites are still down..

This is yet to be fixed, my database VPS is down! and it’s failing to reboot!

I have three websites, all down!! two of them are back online now, but another one (most important one) is still down. I restarted mySQL twice but failed.

Yes, I first looked here and saw nothing.. Customers tweeting was the first indication it wasn’t just me.. I thought this was Simon’s initiative to improve Customer Support?

Our site is still down too- anymore updates?

same situation here, VPS is down

All my MySQL’s are down … Crap! And yes, I know … You get what you pay for … ;-)

My db server is still down. 11:50am CST

All my VPS sites have been down for about an hour now. Should at least get an account credit for the down time.

Shared and VPS MYSQL DBs are all down.

All my sites on echols are showing Apache’s “It works!” default page.

March 10th, 2014 at 9:57 amDuncan Murrell Says:

All my sites, DBs, down for at least an hour. But it’s a good thing this blog is still working! (*sarcasm*)

How much time will it take to be like before ???

I’m having the same issue as Tyrel. 3 different wordpress sites all coming up with the apache “It Works!” page.

Halfback is down for us. Site won’t resolve at all.

Shared and VPS DBs are back up for me.

VPS just now came back up and no more 504 gateway timeout error. Lets hope it stays fixed :-)

All of my websites are still down

All my sites are back up and running.

All of my websites are still down

Still got one VPS down

All my dreamhost sites still down. Any time estimate on when those “some servers” whose run level booted incorrectly will be fixed?

Everything looks fine to me now. Thnxs!

My sites seem to be up now.

My website is still down!

My IMAP email is not working…. is this related?

@Andre: Email shouldn’t be impacted, please contact support if you’re having an email issues.
@Jeff: We’re looking into the database server for your site
@Everyone: It appears the rest of these issues are resolved and your sites should be working

I cant access my websites =S

My website has been down for over an hour! Why is this marked as resolved?

All sites on my VPS have been down more than 2 hours… more than 30 min since I posted a support ticket without a response, and on hold that long waiting for a support chat session. I’d say this deserves an update here.

My website has been down for an awfully long time now, customers are trying to reach the site and I am losing money and credibility with new clientele who are visiting the website

I know it says “Resolved” but I still can’t access my website (which is set to redirect). Pinging it gets no response. I just camped on to Chat but it says there’s a 34 mins. wait!!!!!

Are you sure it has been “Resolved”.

Scratch that last post. Working fine now.

Still having issues with mysql database on my site.

Problems accessing database server still “theseus:thate” – have logged a ticket for the same.

site works… then minutes later it doesn’t… then it works, then doesn’t

Keep getting Error establishing a database connection

I talked to online support… they ‘couldn’t reproduce the issue’.. of course that was when the site wasn’t connecting… now when I just want to check it, it’s giving me an error again!

Site still down.

Cannot connect to Database….

Error establishing a database connection? Just signed up, is this a normal occurrence?

Still down! Over 2 hours and messages sent to support…anyone?

All of my sites are down too because of database server issues. My inbox is being flooded with Uptime Robot notifications that my sites are down, then up, then down again. I cannot access any of my database servers. If I do gain access, it’s extremely slow or I get 500/503 error messages when it times out.

JS: Sadly, yes. From my 2 1/2 year experience, database servers on DreamHost are notoriously unreliable and slow.

@JS & Brian D: Sorry to hear you were still having database issues. There was a single mysql service on the “cookiecutter” server that required further fixing and maintenance after the power issues, but all other mysql services should have been restored within 30 minutes of the reboots.

I personally work on the mysql servers, making sure they are working optimally, and I don’t agree that they are unreliable or slow. They are one of the most reliable services at DreamHost. If you are seeing slowness or issues, please submit a support ticket requesting JustinK look at it with details on how to reproduce the slowness you are seeing, and I would be happy to take a look.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Maintenance for homiemail-sub4 email sub-cluster February 27th 7pm PST (6 hrs) (RESOLVED)
4:41 pm

Scheduled Maintenance for homiemail-sub4 email sub-cluster February 27th 7pm PST (6 hrs) (RESOLVED)

Posted (February 24th, 2014 at 4:41 pm PST) by andrea

On Thursday, February 27th from 7PM-1AM PST, we will be performing maintenance on our “homiemail-sub4″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. During this time, we will also be making backend DNS changes to our mail system. While this shouldn’t cause any customer-impacting issues, we do acknowledge that possibility and will keep a very close eye at all steps during the maintenance. This new solution has undergone rigorous testing and has proven to be effective already on the “homiemail-sub5″ subcluster, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

Please note: Once you have determined that you are on homiemail-sub4 cluster, if you are using outside DNS (ie. you are not the default DreamHost nameservers), you will need to update the IP address for your ‘mail’ A record to the new one: 69.163.253.135.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update February 28, 2014 1:20 am PST: Everything looks great! Traffic is now resolving to the new load balancers, and these changes should result in significant stability improvements to our sub4 mail cluster. We are going to go ahead and close this maintenance tonight. As always, if you notice any issues with your mail services, please contact support! Thanks once again for your patience during this maintenance.

Update February 28, 2014 12:40 am PST: The DNS updates are complete and we are currently monitoring the traffic on the new load balancers. We will update with more information shortly!

Update February 28, 2014 12:10 am PST: Things are still moving along as DNS is updating. As noted previously, you may continue noticing a few connectivity issues; as the DNS changes are completed, traffic will be routed to our new load balancers instead of our older load balancers. Thank you, once again, for your patience!

Update February 27, 2014 11:45 pm PST: Just a heads up – we will be posting less frequent updates unless there is something new to report. As noted previously, DNS records for customers on the sub4 mail cluster are continuing to update. As a result, you may continue noticing a few connectivity issues; as the DNS changes are completed, traffic will be routed to our new load balancers instead of our older load balancers.

Update February 27, 2014 11:30 pm PST: We are still continuing to update all mail DNS records for customers on the sub4 mail cluster. As noted previously, you may continue noticing a few connectivity issues; as the DNS changes are completed, traffic will be routed to our new load balancers instead of our older load balancers.

Update February 27, 2014 10:42 pm PST: We are officially extending the maintenance window to 1:00am (and have updated the post above to reflect this as well). We will continue updating this post as the DNS updates continue, or with any other information we may need to share. Thanks for your patience!

Update February 27, 2014 10:27 pm PST: We’re still in the process of updating all mail DNS records for customers on the sub4 mail cluster and therefore may be extending this maintenance to 1:00am PST to allow for all the updates to complete. We will update this post as the DNS updates continue and to confirm if we will be extending the maintenance window.

Update February 27, 2014 9:36 pm PST: We’re still seeing some connectivity issues with your email due to the DNS changes. Our systems engineering team is looking into expediting any DNS changes. We’ll post another update as soon as we have more information.

Update February 27, 2014 8:55 pm PST: Due to slow DNS changes, you may notice connectivity issues with your email. As DNS changes are completed, traffic will be routed to our new load balancers instead of our older load balancers. We are currently investigating ways to update the DNS faster and will update this post as soon.

Update February 27, 2014 8:00 pm PST: The maintenance has been going smoothly thus far without any interruption of service.

Update February 27, 2014 7:25 pm PST: Maintenance is proceeding as planned and nothing external facing has been changed at this point. We are monitoring as the maintenance progresses. Stand by!

Update February 27, 2014 7:03 pm PST: Maintenance is beginning now with port changes to facilitate networking changes. This should only take a few minutes.

This entry was posted on Monday, February 24th, 2014 at 4:41 pm and is filed under Announcement, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

30 Responses to “Scheduled Maintenance for homiemail-sub4 email sub-cluster February 27th 7pm PST (6 hrs) (RESOLVED)”

My email account stopped connecting via Apple Mail about 45 minutes ago and now gets IMAP errors. Suggestions, other than waiting 4 hours to regain access to it?

When I attempt to connect, it says “Server returned the error: The connection to the server on port 143 failed (error 54: Connection reset by peer.)”

@I.A. We’re not seeing any service interruptions at this time, so it might be unrelated. Would you please send in a support ticket so we can check it out for you?

All my dreamhost hosted e-mail accounts are down right now. 8:25PM PST

Same here. Email is down. But I’m sure the engineers are on to it.

I submitted a trouble ticket, as requested by “Jen D”

All my dreamhost hosted mail is also down and has been for last 30 minutes.

Thanks guys. All up again.

Hi,
Apple mail also not working for me for last hour or so. Webmail works fine.

Same, all Dreamhost hosted email accounts are down.

Mine is still down.

I’m having trouble sending and not getting any incoming mail at all. Support says it’s maintenance per above. Status says “Update February 27, 2014 8:00 pm PST: The maintenance has been going smoothly thus far without any interruption of service.” There is interruption guys… just so you know.

so basically I can’t run my business for 4 hours while you mess around with the server? How about moving the accounts to another server in the meantime?? Or maybe being courteous enough to send an email to me a few hours in advance so I can prepare for it? Well I guess during this blackout I will go looking at other webhosting services like godaddy.

Dreamhost email completely interrupted. Also did not receive any notification about this scheduled maintenance.

Thank you for the week plus notice on the maintenance window! I was able to relay it out in time to customers and though all of the emails are bouncing on connectivity I know things will iron out. I will submit a ticket in the morning if I see issues. Thanks for your hard work and keep us posted.

Yup, what a mess. Constant interruption in service for the past 2 hours+ All I can hope is that this ‘service’ fixes the ongoing problems we’ve been having with email here for a while now. It’s been a crap shoot whether or not emails will send on the first (or third) try without losing server connection or experiencing time outs. Such a constant anxiety inducing drag for the whole team.

@Kristy — I am so sorry for the problems! We are working on getting this back up well inside our planned window!

@Gary — Again, I’m sorry, and hopefully you won’t see this down for much longer.

@Paul — I’m so sorry, and yes, thank you for reminding us to update! It was going so well for awhile.

@RM — I am so very sorry! This post has been up since Feb 24th, although we always hope that our maintenance won’t cause impact. I sincerely apologize that in this case it did.

@Glenn — You’re welcome, and thank you for being patient with us!

@Mel — That is exactly what this maintenance is intended to do — bring an end to the issues we have been fighting. I’m very sorry it has been causing you grief, and hopefully this will be the end of it!

almost an hour and no update…

guess we all just wait patiently until you deem us worthy of wasting your time on telling us when you might fix it… you know – we’re just lowly customers after all.

Hi, the maintenance has caused impact, please can you advise what is happening thanks

I am within business hours in Australia with email that doesn’t work. Not cool. Sorry. I know you’re doing your best, but it is not good enough. It does make me wonder whether I should find an alternative host for my site.

Hi Guyz,

All incoming mail on all domains hosted respond with error :

Task ‘suly@ccctv.co.za – Receiving’ reported error (0x800CCC0F) : ‘The connection to the server was interrupted. If this problem continues, contact your server administrator or Internet service provider (ISP).’

Any eta on when we should be up again?

This has caused my email to go down (POP, IMAP, and SMTP). Webmail is fine however. I apologize for submitting a Support ticket on this – I should have checked here first!

While it’s nice that maintenance is getting mentioned on this status page, I agree with others that it would be a great courtesy to send out an e-mail about these times that there’s maintenance being performed. I’m in the final stages of working on a book and the only time I can communicate with my editors in India is between 7:30 and midnight (when I go to sleep). This basically means I lose an entire day and it’s friday in India, so It’s until Sunday eve. before I can communicate with them again. I’m not angry, just disappointed. This doesn’t engender any feelings of reliability towards the email service provided.

Thanks, DH, for your commitment to continuous improvement! I know these hiccups with mail will lead to better reliability down the road. If I’m having any trouble tomorrow I may submit a ticket after I make the recommended update on the nameserver for the two sites on my account that are registered elsewhere.

Seems to be working intermittently now thanks guyz.

Yeah, not working intermittently or not for me (and I assume many others) still…

They should teach the engineers at Dreamhost about redundancy planning. Rather than taking an entire email cluster offline, they could have moved the accounts temporarily so that the customers don’t feel the brunt of their work. We are in the middle of the business day and our mail servers are down with no idea when they will be back up again.

The Dreamhost customer service motto – “They are all guinea pigs that pay us to screw up their business.”

Problem only to some customers mail adrress.
May you gine me some info to fix all the mail?

Now the problem is only to send mails via SMTP

I had some trouble this morning but averything now is working fine :-)
Many of my domains are not configured with dreamhost nameservers, so I had to modify those ones with new parameters. In next announcements I would like to be noticed about this change (it was a little bit hidden).
Anyway webmails where always working and I suggested my customers to use it for few hours.
Thank you Dreamhost!

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)
7:21 pm

Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)

Posted (February 16th, 2014 at 7:21 pm PST) by Jen D.

We are seeing an increased amount of traffic to our East Coast data center and this is causing connectivity issues to your sites. Our systems and network engineering teams are currently investigating the spike in traffic and working to find the root cause.

We sincerely apologize to those customers who are being impacted by this issue and we will update this post as soon as we have any further news for you. If you are unsure if you are affected, you can check which data center you utilize in the control panel here.

Update 7:53pm PST: We’re not seeing any further connectivity issues within the last 10 to 15 minutes. We have estimated this attack to be a large 30-40 gbps. Since our mitigation efforts, we are not seeing any further issues at this time. We will be closing this post, but our teams will be watching for any further incidents closely through the night. We apologize for any inconvenience this has caused you.

Update 7:31pm PST: We’ve found a DDOS attack was targeting one specific server at our Virginia data center and our network engineering teams have blocked the originating IP address.

This entry was posted on Sunday, February 16th, 2014 at 7:21 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)”

The link to determine which datacenter your server is in does not work.

Oh no not again…..

@Manny — I’m so sorry for the problems, but we seem to get getting a grip on the traffic. Hold on!

@eric: I’m so sorry for the problems! Are you logged into your DreamHost panel? (https://panel.dreamhost.com/) If you are, can you let me know what error you see when you try to visit:
https://panel.dreamhost.com/index.cgi?tree=status.dc&

Thank you for working on it, This explains the issues I’ve been having. . .

can u tell which server was affected? was it a DS ?

@Elizabeth

I thought it was my sites that are having problems and I was constantly monitoring my sites and configuring plugins since last week. I was about to pull my hair. Anyway hoping these DDOS attacks will be resolved as soon as possible and thanks for the immediate actions.

@Elizabeth This is the page I get: http://imgur.com/omIKOcg

Thanks for the quick response, our sites started behaving properly right when the DDoS was mitigated.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West (Los Angeles) Data Center Thursday February 13th 10pm PST (5 hours) (RESOLVED)
7:55 pm

Network Maintenance US-West (Los Angeles) Data Center Thursday February 13th 10pm PST (5 hours) (RESOLVED)

Posted (February 11th, 2014 at 7:55 pm PST) by Brian@dreamhost

On Thursday February 13th, starting at 10pm PST, we will be conducting network maintenance in our US-West (Los Angeles) Data Center. The expected duration of this maintenance is 4 hours. During this maintenance, we will be upgrading the software on our core router. This upgrade requires a reboot of the core. In order to reduce any noticeable impact, we will be moving links, one by one, from the main core router to our temporary switches. During the moves, there will be mostly unnoticeable service impact, lasting no more than a couple of minutes. Once all links are moved, we will then reboot the core. This will be followed by moving the links back to the successfully rebooted core. Comments will be open during the maintenance so that we can assist you in real time, and verify if your issues are related to this maintenance.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.
If you have any questions or concerns please contact the DreamHost Support Team.

Update February 14th 3:10am PST: All links have been moved back to the core and all services are reporting up without any issues. This maintenance is now completely done with no downtime at all. Thank you again for your patience throughout this maintenance. We are now resolving this status post successfully!

Update February 14th 2:00am PST: The upgrade of the core is successful and we will now move the links back to the core. We are going to have to extend this window for one for hour. Thank you for your patience.

Update February 14th 1:00am PST: We are about ready to start the upgrades and the reboots of the core. Sorry for the delay. We have not seen any significant customer impact throughout the maintenance.

Update February 13th 12:00pm PST: Moves are going smooth so far. We are near done with them. Once complete we can upgrade the core and then proceed with the reboot.

Update February 13th 11:35pm PST: We are getting ready to move the links over to our temporary switches now. During the moves, you may notice a temporary service interruption which should last no longer than a couple of minutes. Comments are open so that we can check to see if any reported outages are related.

Update February 13th 11:05pm PST: Moving along slowly to ensure that we do not have any issues. Still no impact expected at this time.

Update February 13th 10:30pm PST: We have not begun to move links yet. We are still in the process of preparing for the initial move. There is no expected service impact at this time.

Update February 13th 10:00pm PST: Our team is together and ready to start on this maintenance. We will update throughout the night as we progress through to the end. We do not expect any major service impact during this maintenance.

This entry was posted on Tuesday, February 11th, 2014 at 7:55 pm and is filed under Announcement, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)
7:01 pm

Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)

Posted (February 11th, 2014 at 7:01 pm PST) by Elizabeth

We are experiencing unusually high traffic to our US-East data center. At this time, we are investigating the cause and we will let you know as soon as we have more information on what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being impacted by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update February 11th, 2014, 7:39 pm PST: The abusive traffic has been blocked and we are no longer experiencing trouble connecting with servers in US-East. If you are still seeing issues, please contact support and let us know so we can help you. Thank you for your patience.

Update February 11th, 2014, 7:26 pm PST:  We have identified and dealt with the source of the issue and are currently checking services to verify that everything is recovering as expected (so far our status reports indicate as much). Thank you again for your patience, and we will update here when we are confident with the resolution status.

Update February 11th, 2014, 7:02 pm PST:  The connectivity issues are due to a massive DDoS targeting our US-East (Virginia) datacenter. Our Network Engineering team is currently working on mitigating that DDoS. Please continue to check here for updates and thank you for your patience.

This entry was posted on Tuesday, February 11th, 2014 at 7:01 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses to “Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)”

all sites are down,gosh…hurry

pff this is not good , please block them .

My website is down, too.
:((

And they told me it was just an issue with cache in my browser…. nice

@a, @adrian, @Kenny — I am so sorry for the problems! We are working on getting everything back up and running as fast as possible!

It’s never a good thing to get a DDoS attack, thanks for letting us know and for resolving it. My sites weren’t affected, but for everyone else that’s having problems it usually takes a day for these to end. And for the guy who said they told you it was the cache, they might have spoke too soon because these kinds of things take time to confirm, not everyone is going to know what’s happening right when it happens, I’ve had nothing but good experience with the support.

Regards,
Jose

We know it ! :D And we will wait for it ! Do your best !

My site is down too :(
I was planning to upload a new site tonight.

If your web and MySQL servers are located in our US-East (Virginia) data center, your sites may be affected by the current situation. Please check your server locations in your panel at Status > Data Centers:

https://panel.dreamhost.com/index.cgi?tree=status.dc

Please reach us here to receive a final update by email and to raise any other related concerns:

https://panel.dreamhost.com/index.cgi?tree=support.msg

Thank you for your continued patience as we work to determine the cause of these issues.

My site is down too :(
lol..I was planning to upload a new site tonight.

My VPS was inaccessible from my home Comcast connection but working from my iPhone’s AT&T LTE connection. How does that make sense?

Later it was inaccessible from both connections but appears to be back online now.

My site come back !
DH is so awesome !!!
I love you ! :|

My site is also down also. So much for finishing that project tonight, shucks.

I don’t know what happened but My site is up again… :)
Thnx Dreamhost for resolving the issue this quickly.

Come on fix the issue just take the servers down.

For me it’s working now

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Wednesday, February 12th, 12am to Thursday, February 13th, 12am PST
7:54 pm

Live Chat Unavailable Wednesday, February 12th, 12am to Thursday, February 13th, 12am PST

Posted (February 7th, 2014 at 7:54 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Wednesday, February 12th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in team meeting, discussing our 2014 plan, as well as specific changes to the teams scheduling and inner policies! We aim to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 12am PST the next morning, February 13th. We thank you for your patience and understanding.

This entry was posted on Friday, February 7th, 2014 at 7:54 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance February 11th 11AM-7PM PST (RESOLVED)
3:41 pm

DreamObjects Maintenance February 11th 11AM-7PM PST (RESOLVED)

Posted (February 5th, 2014 at 3:41 pm PST) by andrea

As part of our continued efforts to improve our cloud services, our Systems Engineering team will be performing maintenance on our DreamObjects service on Tuesday February 11th from 11am-7pm PST. During this maintenance window you may intermittently experience slower upload and access speeds and short periods of connectivity loss. We will be upgrading Ceph, the underlying storage technology that powers DreamObjects. The upgrade will resolve an issue with bucket ACLs not getting saved and another related to large file uploads.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact the DreamHost Support Team for assistance.

Update February 11th, 2014, 12:50pm PST: The maintenance has been progressing as scheduled. As expected, we have been encountering 500 errors and sync disconnects as the upgrades roll out, for which we apologize. We’ll update this post again when we have more information.

Update February 11th, 2014, 3:50pm PST: We are happy to report that maintenance was successfully completed an hour ago. We are now resolving this status post, but if you have any questions or concerns, please don’t hesitate to contact the DreamHost Support Team for assistance.

This entry was posted on Wednesday, February 5th, 2014 at 3:41 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

14 Responses to “DreamObjects Maintenance February 11th 11AM-7PM PST (RESOLVED)”

Thanks for this notification explaining the situation.

If you haven’t done so already could you please announce this through your social channels.

Our whole site has been down during this maintenance period. Why is this being done in the middle of a work day, versus early morning or another low traffic time like other hosting providers?

Kind of cramping our company’s style today, guys.

Cannot send email, is it related to this? When will it be fixed.

Our web site and email is down. To echo another customer’s complaint, why is thus being done at daytime and not on a weekend at night?

WTF is going on there? There is no announced outage or maintenance, 6 days later (Feb. 11), and the email is still fucked up! Can’t access the control panel because it has to email my authorization code and I can’t get on to the mail server to pick up my authorization code. GET IT TOGETHER!!!!

Ditto what others are saying. I’ve been up and down all day with 503 errors. Wouldn’t have been a problem if this happened in the middle of the night or had some warning but causing a lot of problems in the middle of a business day. Support has not been helpful.

February 11th, 2014 at 11:17 amMatthew Williamson Says:

Our email accounts are not working.

East coast/west coast overlap, 9a.m. to 2p.m. and you plop and majorly intrusive maintenance right in the middle of it. That was REAL smart!

No email on my sites. Also no access to panel.dreamhost.com

Is this affecting webmail for everyone? I am getting 503 errors.

It’s time to leave Dreamhost – they are last to the party and always causing interruption. I understand there will never be the ‘best’ time but at least do things like this during non-business hours.

Mail is NOT working for me

@Jamie: You’re very welcome, yes we posted it on our @DHStatus Twitter account. Thanks for the reminder!

@Kevin @George O’Hanlon @John @Chellie @Matthew Williamson: I’m very sorry to hear of the trouble. That particular issue is unrelated to the DreamObjects (cloud storage) maintenance described by this status post; anyone affected by this maintenance is also being contacted directly. We are currently aware of some issues affecting mail servers at the moment, and our admins are working to clear up the delays as soon as possible. Please contact support to receive an update as soon as we have one, if you are indeed affected! Thank you https://panel.dreamhost.com/index.cgi?tree=support.msg

@Kim @Joe @Byron Canfield: Our apologies for the inconvenience regarding the time of this maintenance; we wanted to be sure we have as many on staff as possible familiar with the Ceph technology to help this important maintenance session go smoothly. We aim to keep our maintenance windows minimal moving forward. Thanks for your patience!

@Conciertos en Costa Rica: Please contact us here: dreamhost.com/contact so we can help you regain access to your panel right away.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost Web Panel Currently Experiencing Connection Issues (RESOLVED)
5:40 pm

DreamHost Web Panel Currently Experiencing Connection Issues (RESOLVED)

Posted (January 29th, 2014 at 5:40 pm PST) by Elizabeth

We are seeing some instances of slow connections to the DreamHost web panel (panel.dreamhost.com). Our admins are investigating the root cause, and we will update here when we know more. We sincerely apologize for any inconvenience you may encounter.

Update January 29, 2014, 6:15 PM PST: At this time, all connectivity has been restored. Our administration team is continuing to investigate the root cause to ensure this doesn’t occur again.  If you are still seeing problems, please contact support and let us know!

Update January 29, 2014, 5:53 PM PST:  Our administration team is actively investigating this issue.  We have found that the panel will load slowly or have a connection time out.  This is only causing problems with the DreamHost Web Panel.  Your website, email and hosting services are working correctly.  We are very sorry for the inconvenience caused by this panel interruption.  Please keep your eye on this announcement for continued updates.

This entry was posted on Wednesday, January 29th, 2014 at 5:40 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » NS2.dreamhost.com Hardware Upgrade – IP Change Required for Vanity Name Servers
3:07 pm

NS2.dreamhost.com Hardware Upgrade – IP Change Required for Vanity Name Servers

Posted (January 28th, 2014 at 3:07 pm PST) by andrea

This is a notice to let you know that we are updating the hardware for one of our nameservers, ns2.dreamhost.com. We’re also moving it out of our California data center to the east coast data center to further diversify our nameserver locations. Due to the latter, the IP for ns2.dreamhost.com will be changed to 208.97.182.10 on February 5th, 2014.

Who will be affected by this IP change?

The only customers who will have to actively update their records are those who have their domain registered at another registrar *and* that domain is using glue records for vanity name servers that are pointed to our nameservers.

If you’re unsure if you’re using this setup, a simple WHOIS search will show you your current name servers and registrar.

If this does describe you, you can go ahead and log into that admin interface and make the changes with the new IP address listed above. Even though it’s not completely migrated yet, the new IP address is live and functional for your purposes. You can also just wait until the above date if you wish. There is no super rush to get this done, because after that change, the old IP will be functional for a while longer, but the sooner you make the change the better :)

We will post another announcement when we have a date for removing the old IP address, so keep an eye on dreamhoststatus.com if you can’t get to changing your records soon.

If you have your domain pointed directly to our nameservers, such as ns1.dreamhost.com, ns2.dreamhost.com (and ns3.dreamhost.com, which is optional), or you have vanity name name servers  on a domain that is currently registered with us, then this change does *not* affect you, as we automatically update everything in our system. If you’re unsure if you’re using this setup, again, a simple WHOIS search will show you your current name servers and registrar.

If you have any questions about this change, please contact the DreamHost Support Team.

This entry was posted on Tuesday, January 28th, 2014 at 3:07 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance, Switch Software Upgrades, US-West (Irvine) Data Center Thursday January 23rd 10pm PST (RESOLVED)
3:44 pm

Network Maintenance, Switch Software Upgrades, US-West (Irvine) Data Center Thursday January 23rd 10pm PST (RESOLVED)

Posted (January 17th, 2014 at 3:44 pm PST) by Ralph

On Thursday January 23rd, starting at 10pm PST, we will be conducting network maintenance in our US-West (Irvine) Data Center. During the maintenance we will be upgrading the software on various switches throughout the Data Center. These upgrades will allow us to improve our monitoring capabilities in Irvine. After the software upgrades are complete, we will need to reboot the switches for the upgrades to be complete. These reboots will cause service downtime ranging from 2 to 10 minutes. Services affected will be FTP/SSH and access to your website. Mail will not be affected by this maintenance. We will proceed with these reboots slowly in the event that any hosting machine needs direct attention. This maintenance will mainly affect customers on dedicated server hardware. There is only one switch that is the home of shared and VPS customers, the rest are dedicated machines. We we will make every attempt to announce which machines are going to be affected as we progress through the maintenance, so please keep an eye out for our updates. When the maintenance starts, comments will be open so that we can assist with you any possible issues that may come up after the reboots.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center. During the maintenance, we will do our best to let you know exactly which machines will be affected and when.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 24th 12:40pm PST: All 9 public reboots and the 6 private network reboots have been completed successfully with very low impact on server availability. Thank you again for your patience throughout this maintenance. We are now resolving this status post successfully!

Update January 24th 12:30pm PST: We are approaching the light at the end of the maintenance. :) 8 out of 9 of the reboots are now complete. The final reboot is starting now and the list of machines affected by this reboot has been added to the comments. Thank you for your patience throughout this maintenance.

Update January 24th 12:20pm PST: 7 out of 9 complete. This one went really smooth. Two more to go. Next group of machines with service impact from this reboot is now added to the comments.

Update January 24th 12:10pm PST: 6 out of 9 complete! The service interruption for that reboot was around 10 minutes. We are moving on to the next and the server names have been posted for this as well.

Update January 24th 12:00pm PST: Previous post was sent a tad too early. We were checking that everything in the previous reboot came up correctly still. We did start the 6th reboot now and that is requiring a hard reboot as well. We should have service restored in a few minutes.

Update January 23rd 11:40pm PST: That reboot completed without anything major occurring. Lasted roughly 4 minutes. We are 5 out of 9 reboots complete. The next list of machines have been posted.

Update January 23rd 11:35pm PST: We are 4 out of 9 complete now. The next one will be the one switch that connects to both shared and VPS machines. These machines are now listed in the comments. We unfortunately can only list host machines, not the actual VPS guests, or the list would be very large. Thanks for your patience through this maintenance!

Update January 23rd 11:20pm PST: Third reboot is now complete. That one went smooth and was complete in 3 minutes. Next group of machines are now posted in the comments.

Update January 23rd 11:15pm PST: We just completed the second reboot out of 9. This one hit a bit of an issue that required a hard reboot, or power cycle, of the switch. Service was interrupted for around 15 minutes. We are now moving on to the third reboot. Machines have been posted in the comments.

Update January 23rd 10:55pm PST: First public switch reboot is complete. We estimate at around 4 minutes of service interruption. All machines connected came back up without any noticeable issues. We are moving on to the next switch. Comments have been updated to include the next group of servers affected.

Update January 23rd 10:50pm PST: We have completed the private reboots without any incident. We are now ready to proceed with some public network switch reboots. This will have service impact for the machines connecting to that switch. The machine names are listed in the comments of this post.

Update January 23rd 10:15pm PST: The first few reboots have completed successfully. As a reminder, these reboots we just did affected the private side of our network, so there would be no visible service interruption during this stage of the maintenance.

Update January 23rd 9:58pm PST: Our team is ready to begin the maintenance. We will be starting the reboots on switches that will not affect your hosting service. This will allow us to run some further tests to make sure everything will go as smoothly as possible. Comments are now open, so if you notice any issues please let us know.

This entry was posted on Friday, January 17th, 2014 at 3:44 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Network Maintenance, Switch Software Upgrades, US-West (Irvine) Data Center Thursday January 23rd 10pm PST (RESOLVED)”

ds1922, ds5774, ds5217, ds1919, ds3124, ds1917, ds1916, ds5030, ds6460, ds6563, ds1909, ds5131, ds1907, ds6294, ds1905, ds6217, ds6545, ds2753, ds1900, ds5240, ds1893, ds3823, ds6642, ds1896, ds6159, ds5809, ds1886, ds1885, ds3821, ds6292, ds3929, ds3822, ds2287, ds6376, ds5808, ds1892, ds2290, ds5739, ds2103

ds4145, ds3826, ds6676, ds5096, ds5197, ds5669, ds1932, ds4491, ds6546, ds6297, ds3827, ds6564, ds4952, ds6330, ds3718, ds1960, ds5172, ds6448, ds6363, ds1956, ds6096, ds4612, ds5479, ds1952, ds6316, ds6255, ds6130, ds5218, ds5887, synapse, ds3352, ds3265, ds1948, ds1949, ds6386, ds6295, ds5063

 digicm, ds3513, ds3512, ds4621, ds3744, ds4205, ds6323, ds3899, ds6679, ds3898, ds4865, ds5150, ds3896, ds5816, ds4538, ds6109, ds5903, ds5902, ds4074, ds5540, ds3495, ds6103, ds5624, ds3891, ds3491, ds5082, ds4166, ds3757, ds3919, ds4677, ds5795, ds5846, ds5958, ds6524, ds5834, ds6639, ds6572, ds6523, ds6273

ds6195, ds4003, ds5577, ds5491, ds2206, ds3245, ds1673, ds6253, ds6329, ds1671, ds3966, ds6078, ds5378, ds5743, ds5556, ds5704, ds5456, ds2085, ds6599, ds1679, ds1687, ds1686, ds6656, ds5579, ds1681, ds3811, ds4388, ds4589, ds1731, ds6062, ds6457, ds4654, ds1715, ds1718, ds6626, ds6512, ds6332, ds3256, ds5555

homie-vserver360 / homie-vserver361 /homie-vserver362 / homie-vserver363 / homie-vserver360 / homie-vserver361 /homie-vserver356 / homie-vserver357 / homie-vserver362 / homie-vserver363 / homie-vserver358 / homie-vserver359 / homie-vserver356 / homie-vserver357 / homie-vserver352 / homie-vserver353 / homie-vserver358 / homie-vserver359 / homie-vserver354 / homie-vserver355 / homie-vserver352 / homie-vserver353 / homie-vserver348 / homie-vserver349 / homie-vserver354 / homie-vserver355 / homie-vserver350 / homie-vserver351 / homie-vserver348 / homie-vserver349 / sandalwood / santamaria / homie-vserver350 / homie-vserver351 / sapele / sassafras / sandalwood / santamaria / poplar / ramin / sapele / sassafras / redheart / sal / poplar / ramin / homie-vserver369 / homie-vserver370 / redheart / sal / homie-vserver371 / homie-vserver372 / homie-vserver369 / homie-vserver370 / mpingo / oak / homie-vserver371 / homie-vserver372 / mpingo / oak / padauk / palisander / ipe / aspen / padauk / palisander / greenheart / walnut / ipe / aspen / iroko / maple / greenheart / walnut / marblewood / meranti / iroko / maple / marblewood / meranti

ds5683, ds5779, ds5894, ds5763, ds4037, ds5208, ds5123, ds6680, ds3542, ds5625, ds4721, ds3413, ds6374, ds5207, ds6373, ds5013, ds5176, ds5626, ds3532, ds6191, ds5426, ds3766, ds3820, ds5235, ds6065, ds3763, ds6426, ds6486, ds6110, ds6641, ds4120, ds3520, ds5119, ds3518, ds6547, ds5120, ds6422, ds5083

ds5515, ds6652, ds5644, ds5596, ds5643, ds6208, ds6207, ds5821, ds6525, ds5638, ds5637, ds6307, ds5635, ds6250, ds5633, ds6147, ds5819, ds6206, ds6306, ds5628, ds4861, ds5627, ds4640, ds4639, ds5088, ds4813, ds4948, ds4811, ds4810, ds6427, ds5012, ds5818, ds5427, ds6305, ds4628, ds6190, ds4626, ds4625, ds4453

ds5693, ds6559, ds5685, ds5682, dedicated, ds6155, ds5684, ds6423, ds6287, ds5687, ds5764, ds5526, ds5661, ds5728, ds6081, ds5604, ds6640, ds5523, ds6368, ds5820, ds5601, ds5658, ds5657, ds5656, ds5655, ds6428, ds6324, ds6263, ds5652, ds5651, ds5650, ds5518, ds6390, ds5648, ds5647, ds6148, ds5646, ds5645, ds5598

ds6666, ds3414, ds5114, ds3412, ds4082, ds5710, ds6628, ds3426, ds3418, ds5226, ds4598, ds3423, ds5622, ds5227, ds3747, ds5388, ds5621, ds6165, ds3742, ds3741, ds5723, ds3795, ds5175, ds5677, ds6607, ds3440, ds6485, ds6242, ds5233, ds6341, ds6648, ds4614, ds4943, ds4830, ds4017, ds3460, ds3878, ds6205, ds6261

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance US-West (Los Angeles) Data Center Thursday January 16th 10pm (4 hour) (RESOLVED)
11:32 am

Network Improvement Series Maintenance US-West (Los Angeles) Data Center Thursday January 16th 10pm (4 hour) (RESOLVED)

Posted (January 15th, 2014 at 11:32 am PST) by Ralph

As some of you know, we have been working very hard to improve our network over the past 2+ years with our Network Improvement Series. On Thursday January 16th, in a 4 hour window starting at 10pm, we will be moving on to the next step of the Network Improvement Series maintenances. This step is going to involve migrating the network layer from the old hardware to the new designed setup. This is expected to be a near impact-free maintenance; all services will continue to work throughout the migration. As with all maintenances, we will proceed slowly to ensure that, should any problems occur, we will be able to address them quickly. Comments will be open so that we can address any problems you may be experiencing. We are excited to continue and eventually conclude the major network overhaul we’ve been working on to improve your hosting experience. We thank you for your patience through the years.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 17th 1:48am PST: We’re happy to report that this step of our network maintenance series is now complete. We are now on the new hardware, and the process was completed without any customer impact. Keep an eye out for upcoming maintenance announcements, and as always if you have any questions or concerns, please contact the DreamHost Support Team.

Update January 17th 1:08am PST: All steps of the maintenance as it relates to customer services have been completed. We’re currently in the cleanup phase and will resolve this post when that’s complete. Thank you again for your patience. We’re closing the comments, so if you have any questions or concerns, please contact the DreamHost Support Team.

Update January 17th 12:38am PST: We have successfully completed the customer-facing part of the maintenance, without any noticeable effects. We’re moving on to the next steps and getting closer to completing the process. We’ll update this post again when we have news.

Update January 17th 12:20am PST: We are ready to move forward with the maintenance. This next step may cause some mail connection issues and delays in delivery, but we’re keeping a very close eye to make sure it goes as expected and can react as soon as we see graver than expected effects.

Update January 16th 11:37pm PST: Just a quick update to let you know we hit a snag while confirming the success of one of the steps in the maintenance. It’s not a customer-facing issue, so there is no service interruption, but it may extend our maintenance a bit. This is just one of the reasons why we leave a more-than-generous maintenance window to complete all the steps with all the necessary checks and confirmations. We’ll update again when we have news. Thanks again for your continued patience.

Update January 16th 10:45pm PST: The maintenance is moving along as expected, so far without impact on any of our services. We’ll update again within the hour or sooner if necessary to let you know how the process is going.

Update January 16th 10:14pm PST: We are starting the maintenance and will be updating this post with the milestones and any other necessary information. We appreciate your patience while we go through the steps and apologize for any inconvenience it may cause.

This entry was posted on Wednesday, January 15th, 2014 at 11:32 am and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West (Irvine) Data Center Friday January 10th @ 10:45pm PST (RESOLVED)
7:01 pm

Network Maintenance US-West (Irvine) Data Center Friday January 10th @ 10:45pm PST (RESOLVED)

Posted (January 10th, 2014 at 7:01 pm PST) by Sean

Due to the recent connectivity issues experienced on January 7th in the US-West (Irvine) data center and some changes made in our related maintenance last night, our team has decided it is best to conduct another maintenance as soon as possible on our network gear to help prevent future connectivity issues that could lead to downtime. Part of our maintenance plan includes moving links to temporary switches, then back again to the core. When working with the network, even with redundant distribution, there is always the risk of some service impact. Due to this risk, we will be monitoring the network, comments and support requests to ensure that we keep you informed and are able to adjust our processes throughout the entirety of the maintenance window.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 11th 02:12am PST: The network maintenance has been completed!

Update January 11th 02:12am PST: We are moving all links back to the core from the temp switches. So far so good.

Update January 11th 01:50am PST: Everything is looking real good at the moment. IPv6 connectivity has also been restored. The maintenance is still moving along and we’ll provide another update soon.

Update January 11th 01:22am PST: The network maintenance is still ongoing. We have not seen any issues but are continuing to monitor everything closely.

Update January 11th 01:00am PST: Changes being made right now will impact account changes in the admin panel. Otherwise, you should not see any issues with your service. We should be finished with this part soon so changes will no longer be delayed.

Update January 11th 12:43am PST: It’s been smooth sailing for this network maintenance. This is just an update to let you know everything is still going well.

Update January 11th 12:18am PST: Everything is moving along smoothly as we continue to move links to the temporary switches. No services have been interrupted.

Update January 10th 11:42pm PST: The network maintenance is progressing as planned. We are currently moving links over to the temp. switches. Everything is looking good and we have not found any interruption of services.

Update January 10th 11:17pm PST: We are now beginning the network maintenance. There should not be any interruption of services.

Update January 10th 10:50pm PST: We are getting ready to begin this maintenance soon.

This entry was posted on Friday, January 10th, 2014 at 7:01 pm and is filed under Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency Network Maintenance US-West (Irvine) Data Center Thursday January 9th 10pm PST (RESOLVED)
10:59 am

Emergency Network Maintenance US-West (Irvine) Data Center Thursday January 9th 10pm PST (RESOLVED)

Posted (January 9th, 2014 at 10:59 am PST) by Ralph

Due to the connectivity issues experienced on January 7th in the US-West (Irvine) Data Center, our team has decided it is best to conduct maintenance on our routing network gear to help clear up a lingering issue with IPv6 accessibility. Our maintenance plan will enable us to keep all services active while we work towards correcting IPv6 support at the US-West (Irvine) location. When working with the network, even with redundant distribution, there is always the risk of some service impact. Due to this risk, we will be monitoring the network, comments and support requests to ensure that we keep you informed and are able to adjust our processes throughout the entirety of the maintenance window.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 10th 1:20am PST: This concludes tonights maintenance. We will have another announcement once we are ready to attempt our next maintenance to correct IPv6 issues in US-West (Irvine).

Update January 10th 1:00am PST: The good news is that we handled this maintenance with no noticeable impact to service. The bad news is that IPv6 functionality has not been restored yet. We do need to complete the rebalancing, but IPv6 functionality will require us to have another maintenance. We are confident we know the solution, we just need to get the planning and preparation fully complete before moving further on troubleshooting. Thank you for your patience.

Update January 10th 12:55am PST: Unfortunately the reboots did not fix the issue with IPv6. We are attempting one final step to see if we can get this working. There will be service interruption to the Us-West data center. We will do our best to minimize this to less than a minute.

Update January 10th 12:50am PST: We are back online, now we have to balance the network back to the rebooted part of the network. This shouldn’t take too long if everything goes smoothly. We will then test IPv6 to ensure it is working correctly.

Update January 10th 12:40am PST: We got past that last hurdle and we are now proceeding with the second and final reboot. Thank you for your patience.

Update January 10th 12:15am PST: We are currently paused on this maintenance. We have hit a bump that we were able to roll back but require more hands to help us through a portion of the maintenance that concerns DreamObjects. There is no service impact, we are just waiting for a few more hands to help assist us through this part of the maintenance.

Update January 9th 11:50pm PST: We are nearing the second reboot. Everything is moved over with only seconds of service interruption. As soon as the second reboot starts we will update this post. Once reboot is complete, we will move everything back to it’s previous state and start our tests on IPv6.

Update January 9th 11:40pm PST: Everything is still humming along smoothly. We are still not quite at the point of reboot, but we are making progress towards that.

Update January 9th 11:25pm PST: This is proceeding as the previous moves we did. Any issues we bump to, we immediately roll back, troubleshoot, than attempt again. So even though we may hit a few bumps in the road here and there, we are still progressing nicely. We are not ready for the second reboot just yet.

Update January 9th 11:10pm PST: We are continuing and hit a minor issue that we were able to roll back quickly. We are investigating before proceeding any further.

Update January 9th 10:55pm PST: Reboot is complete. We will now be moving traffic back over this portion of the network. We will move it slowly to ensure it’s all working as expected. Once everything is moved, we will attempt the second reboot needed tonight.

Update January 9th 10:45pm PST: We are beginning the reboot now. This portion of our network hardware is not currently being used, so we do not expect any issues. Once the reboot is complete, we will need to do this process again for another portion of the network.

Update January 9th 10:40pm PST: We are still not ready for the reboot but we have not had any service impacting issues to this point. We will be ready for the reboot fairly soon.

Update January 9th 10:20pm PST: Everything is moving along smoothly currently. We are still in the process of moving things before the reboot.

Update January 9th 10:05pm PST: We are currently getting ready to reboot a portion of the network. It is redundant, but we are slowly shutting things down to make sure we catch any problems as fast as we can prior to rebooting.

Update January 9th 10:00pm PST: Our team is gathered and ready to begin work on the maintenance. Our support team will be monitoring comments and incoming support. We will keep you updated as often as we can about the progress of this maintenance. We expect that no services will be impacted, but we are standing by and watching closely.

This entry was posted on Thursday, January 9th, 2014 at 10:59 am and is filed under Data Center Maintenance, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat to be turned off for short internal training sessions on January 9th and January 15th (RESOLVED)
2:50 pm

Live Chat to be turned off for short internal training sessions on January 9th and January 15th (RESOLVED)

Posted (January 8th, 2014 at 2:50 pm PST) by andrea

In order to further improve our Live Chat customer experience, we are conducting training sessions for our team members on 1/9 at 12pm noon and at 4pm PST and on 1/15 12:15am PST. These are short, up to one hour, sessions each but we deemed it necessary to turn off Live Chat for the duration. You will still be able to contact our tech support team by submitting a ticket. We apologize for the inconvenience.

This entry was posted on Wednesday, January 8th, 2014 at 2:50 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues in US-West (Irvine) Datacenter (RESOLVED)
7:18 pm

Connectivity issues in US-West (Irvine) Datacenter (RESOLVED)

Posted (January 7th, 2014 at 7:18 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Irvine) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update: January 7, 2014 10:11pm PST: At this time, all connectivity has been restored and all internal alerts are back to normal. We are resolving this issue for now and will continue to investigate the root cause to ensure this doesn’t pop up again.  If you are still seeing problems, please contact support and let us know!

Update: January 7, 2014 9:51pm PST: We are holding off from resolving this status post while we continue to troubleshoot the issue. At this point, all connectivity has been restored, and your sites should be resolving. We are monitoring all alerts to ensure we do not lose connectivity again. Thank you for your patience!

Update: January 7, 2014 9:09pm PST: We have restored connectivity through the Irvine border, and are no longer routing through the Los Angeles data center. We are still investigating the root cause, and we appreciate your patience as we continue to work!

Update: January 7, 2014: 8:52pm PST: We are still currently working on rerouting the traffic through our other US West (Los Angeles) datacenter and will provide you with an update on its progress as soon as possible. We appreciate your continued patience.

Update: January 7, 2014: 8:32pm PST: We’re still working on rerouting the traffic through our other US West (Los Angeles) datacenter in order to help alleviate the connectiviy issues while we work to identify and correct the problem in this datacenter. We’ll provide you with an update again soon with details on our progress.

Update: January 7, 2014: 8:12pm PST: We are continuing to investigate the root cause. However, we believe that it may be caused by our border router in this datacenter,  and are currently working on redirecting all traffic to go through our router in our other US West datacenter (Los Angeles). Thanks again for your patience.

Update: January 7, 2014: 7:46pm PST: We have found that the connectivity issue is affecting all of the US-West (Irvine) Datacenter and are still looking into determining the cause. We appreciate your patience and apologize for the inconvenience.

Update: January 7, 2014: 7:34 PST: We are still investigating the cause of this issue, thank you for your patience!

This entry was posted on Tuesday, January 7th, 2014 at 7:18 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

149 Responses to “Connectivity issues in US-West (Irvine) Datacenter (RESOLVED)”

Wow. I could not believe this. Was in the process of updating my site and was actively watching a visitor spend a huge amount of time looking for what was about to go live. Then connection is lost. Gone. Well, thanks, that’s awesome. Hopefully I will still be able to pay rent this month with this lost client.

How do I know if my server is in the US-West DC? I need to know if I need to place a ticket as my domain is currently not responding to anything.

Thanks,

-Chris

Yep. All my websites are down. This is the third time in 12 months that this has happened. Is this normal? Or is it time to switch hosts. Perhaps someone with more experience in this area can chime in re normal service commitments. Thanks.

Please try to fix this issue ASAP.

site down !!
How long will it take to get fixed??

Looking at the recent posts, it seems Irvine is having most of the issues these past few months, why?

Yeah I just had a lot of sites go down. And the phone is ringing. I’ve been a long time customer, but now I’m thinking it’s time to move. I presume you offer this service for free seeing as you’re currently not providing me the service I paid for?

@ Chris Todd: In your panel, look under Status> Data Centers. The direct link is in the post: https://panel.dreamhost.com/index.cgi?tree=status.dc&

Third time in the last year sounds about right. What’s ironic is that I was going to look at some of my WordPress settings to start prepping for a move. DreamHost has way too much down time to be worth what they charge.

How long do you think this will take? my website is down!!

I second that Irvine has had the most problems. Seems to always go out at the worst time too. One time last spring it was down for nearly the entire day. I think I need to switch. Any recommendations?

I need to launch my website…

I am also unable to log in to the control panel to see if I am affected, I am getting an error message saying that my login details are incorrect. Is this normal??

Please could you advise is the coralbells.dreamhost.com server is affected?? Please fix ASAP!

@Doug Hatch: I am so sorry for the problems! This isn’t normal, I swear!
@Boringly: Definitely, our goal is to fix this as soon as possible!
@diego: I don’t know yet how long it will take to fix, but we’re working on it right now!

Just set up Pingdom today, and sure enough it is already reporting my website is down… :-(

@Chris Todd > Check under your account > Status: Data Centers

We are facing the problem too. We moved our servers from LA to Irvine for similar problem. Now Irvine does the same.

We are a small nonprofit focused on violence reduction and just had a traffic breakthrough yesterday and today. Please hurry.

The only way this will change is if we all insist that Dreamhost spend some money and implement some redundancy. It’s atrocious that they don’t have it. When I was sold the service (face-to-face at a convention), I was assured there was “zero down-time.” We have to insist upon redundancy, or this will continue to happen.

I currently have 70+ sites down.

Downtime sucks :(

Great way to start my work year folks!

I only have four sites down, but one just went live recently and I have months of work to do. I do agree that there should be backup servers for these kinds of failures. Not cool, DreamHost. You guys need to step up to the plate and bring on an A-game.

Here we go again :D

I heard that the Irvine data center has problems a few times a year, because it tries to skip from M1 to R2, and skips M2, M3, and R1. The rapport phase of website hosting must be approached carefully, because M2 and M3 are important.

Also, you have to follow the 3 second rule when maintaining web servers. Web servers need constant validation, and in order to provide that validation, your status as a data center maintenance person must be above that of the web server, otherwise you will have to use DHV’s to get the web server back on track.

If rebooting the servers doesn’t work, try negging the main target server, then roll off. Also during the time when the web servers are working correctly and there are no problems, experimenting with improving the midface of the servers could result in more long term stability.

2 VPSs down – approx 20 sites affected.
Glad I managed to move 8 sites over to AWS today – they are at least up.

Absolutely frustrating. I switched to a VPS after my site was down for overloading bandwidth even though they say bandwidth is unlimited. And now the VPS isn’t even loading. Will be switching hosts once it’s back up.

@Nicole: You should still have access to panel.dreamhost.com – please try again or reach us at dreamhost.com/contact if you need help gaining access. Your server ‘coralbells’ may indeed be affected by this issue, however. Please watch here for updates!

This would explain the intermittent issues I’ve been having for the last couple of months.. Was seriously thinking of lodging a support ticket today and then this happens. All my sites are on the same server. I’ve always recommended this service extremely highly.. Hopefully they can fix this fast of us to a new data centre with better uptime.

Are objects down too? I’m having trouble accessing mine via the panel

3 hours later still all my sites are STILL DOWN – what is GOING on (you disabled Tech Support Chat, which recently the person(s) I have been getting are clueless & not helpful at all) — WHEN ARE MY SITE and CLIENTS SITES COMING BACK ONLINE THIS IS CAUSING LOST OF INCOME FOR ALL PARTIES INVOLVED PLEASE GIVE A MORE RECENT UPDATE Thank you !!! [will we be receiving a refund this month!?)

Any word yet? Maybe its the same thing as the last time – with the power rack and the thing… See last outage report.

First started with this error
#2013 – Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110

After all was down

Fix it please,

From Dreamhost’s website:

—-
“DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!”
—-

‘Will credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof)’, it says. What this tells me is one may experience 10 hours of downtime, but not get 10 days free – since the mighty caveat has been uttered: or fraction thereof. It does not state who makes what decision to credit less or more – only that it could be a fraction thereof. Ergo: Experience 10 hours of downtime and get potentially 1 day free credit. Though not false advertising, it’s certainly a kind-of bait-and-switch.

Then it tells us there’s no hassle to get this credit to the account. But let’s be clear: the free days for hours of interrupted service is *not* automatic. We have to *submit a support ticket*, or the offer is *invalid*.

Then it says it’s no hassle: To who? Dreamhost or the Customer?

In other words, the Dreamhost’s uptime guarantee simply isn’t. It comes with too many strings attached, too many caveats, stipulations and the very thing they said it wasn’t: Hassle.

The second time, dreamhost!
If you have another one, I swear I will drop my hosting plan and go somewhere other…
This is unacceptable.

♪♫ Save my life I’m going down for the last time! ♪♫ Head East ~ Never Been Any Reason

Our sites are all down as well. And I definitely think about three this year so far sounds right. Quite frankly this is ridiculous.

Sorry folks my fault, keep forgetting of the time zones (scratch the 3 hours in my previous comment), but this is very frustrating, my clients phone calls aren’t pretty when his sites go down…

Yup, blackout on our end.. We’re working on a major blogging platform and can’t afford any downtime when we go live. Doug mentioned three blackouts in the last year; That is acceptable (if the facts are correct). Life happens, just get up back up as quickly as possible.

It is a little inconvenient, but it hasn’t even been a half hour yet. I have been with Dreamhost for well over a decade. A few minutes here and there isn’t a big deal. I doubt any of my clients will notice *cross fingers*. I am definitely not going to bust a vein over this. Dreamhost in general has gotten a lot better over all in the last few years. My experience with other providers has been dismal generally. The only thing that is down for me is MySQL. That will only be a big deal if it lasts a long time.

Can we ask for refunding ? Dreamhost can’t do this time and time again…

Such anger!
Much fail!
No redundancies!
Wow!

I want my website to be hosted on the server where DreamHostStatus.com is hosted. Then I’ll never have any problems.

Pretty sure there’s an issue with objects too… Can’t view objects or create users

In my experience, DH has gotten much worse over the past few years. I’m on a dedicated server and this is the third time in the past 4.5 months. This is well below the level of service I expect for the price I pay.

My dedicated server is down. I pay extra for a dedicated server so I shouldn’t expect these kind of problems…

Complaining here will do NO GOOD. Dreamhost does not look here. THE ONLY WAY to make a difference is to write in and insist on redundancy. If it continues to only be me whining about it I am just that lone crazy chick out in NorCal screaming about redundancy.

If they are inundated with client insistence they might respond.

“I want my website to be hosted on the server where DreamHostStatus.com is hosted. Then I’ll never have any problems.”

+1

The issue started for me at 6:50pm and indeed with the error:
#2013 – Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110

I guess my 3 dedicated servers are in that “small subset”?

you get what you pay for…. if you want physical redundancy buy it. They sell it.

MatchBrook, you’re reading it incorrectly. The “fraction thereof” refers to the hour, and indicates that any portion of that quantity yields the whole—it’s rounding up. So, in this case, “1 day hosting free for each 1 hour (or fraction thereof)” would mean that any couple minutes of downtime would yield a free day, 90 minutes 2 days, and so on.

Apparently I “owe” Dreamhost $27 – Well I just lost upwards of $300+ in sales, how much do you owe me Dreamhost?

I want to switch. Elizabeth, can you please tell me how? And who do I speak to regarding fees being waived?

Well, I already decided to move to another host by the end of the month. You guys are growing too quickly and don’t know what you’re doing. I’ve had more outages than I can handle, especially for the price. Sorry!

Been with DH for over 10 years too, finally decided enough is enough, moved to FireHost six months ago, and not 1 second of down time. Our main sites are already moved, guess I better get on the ball and move the rest off of DH.

Also, it’s 2:45pm in Australia – this is peak time. What is going on???

So sad how much money I’m losing…

Getting a little old.

It’s my fault! Every time I blog, Dreamhost has an outage. I didn’t even get publish the post! Fortunately, WordPress 3.8 takes advantage of local storage, so I didn’t lose my work.

Dreamhosters, it’s a *jungle* out there…

As a reminder: to see which data center your web and MySQL servers are in, please visit Status > Data Centers in your web panel here:

https://panel.dreamhost.com/index.cgi?tree=status.dc&

Please feel free to reach us at Support > Contact Support as well, and we will be sure to send you an update via email as soon as we have more details!

https://panel.dreamhost.com/index.cgi?tree=support.msg&

All my sites are down. How long does it takes to up and running? This is happening second time in this month for me.

ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.
ALL WORK AND NO PLAY MAKES JACK A DULL BOY.

F5

Don’t forget what you pay for the service… It’s very inexpensive!

It’s really not that inexpensive.

Not inexpensive for us on a dedicated server

Just put in a ticket request based on their uptime guarantee policy. No need to get inflamed about this. I’ve been with Dreamhost for something like 7 years, and as a developer having used other hosts, Dreamhost is very good for the money. They’ll get it fixed. If you lose clients over this, your clients are a little too demanding. Errors happen.

Cloudflare can help with these situations, as will Pingdom which gave me an instant alert that things were down. Prepare for issues because they will happen. It’s not if, but when.

Thank you Dreamhost for all your efforts.

@Chris – The post provides the information you need. If you visit https://panel.dreamhost.com/index.cgi?tree=status.dc& you can see where your various servers are (for example your mail server and web host may be located in different locations).

I can’t speak for Dreamhost, however Irvine is close to Brea (main offices) in Orange County. In Invine (and DH’s 2 other SoCal locations), the datacenters are run by Alchemy Communications (of which DreamHost is the majority owner). They have had to expand at an incredible rate and so problems ensue (a little edit here can bring down the whole shebang). Their servers in Ashburn, VA (a great hub for datacenters BTW) are under the care of RagingWire, which opened its doors in Q2 of 2012 I believe. Behind the scenes a lot of things are happening to improve service but of course when things go right, they go right… when they go wrong, people get mad!

It’s 11pm on the east coast and I just want to un-schedule my 6am post. It’s not ready and I’m not going to get up at 5 am to check it. PLEASE bring up ASAP! It’s been down for over an hour.

Update: If your web and MySQL servers are located in our IRVINE data center, your sites are affected by the current situation. Please check your server locations in your panel at Status > Data Centers:

https://panel.dreamhost.com/index.cgi?tree=status.dc

Please reach us here to receive a final update by email and to raise any other related concerns:

https://panel.dreamhost.com/index.cgi?tree=support.msg&

Thank you for your continued patience as we work to determine the cause of these issues.

Also, at first when I got my Pingdom alert, I thought maybe it was something related to the polar vortex. Well, since it’s Cali, no go. I’m in AZ and I’m about to head outside to our pool at 9 pm our time to relax, which is heated, and soak for a bit. Cheers!

An hour? Really? Can us poor long suffering Irvine CA datacenter of disaster get a credit please?

@Tony

Well, that’s the nature of business, people expect things to go right, that’s what we pay for. And if they don’t, people elsewhere.

As a customer of Dreamhost for ~7 years, I pay good money and shouldn’t have to suffer outages because you guys are trying to grow your business too quickly without the right infrastructure to back it up.

When you pay $3.95 – $8.95/mo for basic hosting, it’s incredibly inexpensive for what you get. As far as dedicated servers…. agreed – it should not be happening, especially to you.

“The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!

How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!”

From http://www.dreamhost.com/servers/vps/

So you have to open a support ticket in order to get credit? What if you don’t notice that your site is down for a couple of hours?

I’m trying to access my panel but can’t retrieve my lost password…email to retrieve isn’t arriving…any help here please?

I don’t even want the credit. Like said before… Its really not that expensive. I want uptime. A few hours a year is nothing but last time it was down something like 6-8 hours. Id rather pay more. Other hosts are faster so I don’t want to hear the inexpensive argument. I already get what I pay for when the site is up- Now that its down, Im truly NOT getting what I pay for.

well that makes the third time my site has gone down in a month. and i have the screenshots to prove it.

meh. prolly just a minor NSA gear install. we’ll be back up and transparent proxy’d in no time. DH has done better than most in holding out; but the transparent society is much more inevitable than the zombie apocalypse. lol. http://en.wikipedia.org/wiki/Proxy_server#Detection AWS goes down too. nowhere is safe or certain. learn to surf the ambiguity, or be crushed by its quantum waves. heh.

If you are moving, please check this thread:

http://www.webhostingtalk.com/showthread.php?t=1335589

Stay here with us, it could be worst my friend

Whats the use ranting about it? These guys dont bother about anything we write. In the end we will accept it and get back to work. Have some whiskey. Will help ease the pain :)

Yep, all my sites are down. Why am I paying for a dedicated server again?

I have been a DreamHost customer for over ten years. I can honestly say that for the first eight years or so, I never had downtime. There have been semi-regular outages, and larger issues, which previously were not common. Support still responds, but not as thoroughly or as quickly as in the past. I am still happy with my service, but I do feel service levels are lower, and could be improved upon. Redundancy is critical. All of my websites are currently down.

I have customers who’ve spent significant money with me this week calling, concerned that my site is now gone, worried that they’ve been taken. You don’t have redundant servers to cover us, your customers, in the event of outages like this? Yet another example of a major “FAIL” on your part Dreamhost…

I want to be on the same server of http://www.cam4.com , is always on :) :) :) :)

Again! I am in the process of moving to Host Gator. These pop up issues of site(s) down time happens more and more and more. Can not afford my money maker sites to be down. Not acceptable.

Limited number? Seems that is a bit of an understatement !

Tell me when it comes back up, hopefully writing something here will do that, eh?

I’ll bet they have redundant servers. Dedicated, no less!

This place only exists so we will have something to mutter over while other people keep asking us, “what’s up with the website, man?” A distraction, no more.

“The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!

How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!”

From http://www.dreamhost.com/servers/vps/

So you have to open a support ticket in order to get credit? What if you don’t notice that your site is down for a couple of hours?

i want to be on whatever host this website is ;P

Rule #1 When all connectivity is gone. Check your core routers. This approach has been working for many years.

10%? Thanks for saving me $20 even though I just lost a few hundred by my site being down…

Update: All servers are up, but our border router at Irvine is affecting all inbound/outbound traffic. We are redirecting traffic through our other US-West data center in Los Angeles, which should have all sites resolving again while we continue working on the router at Irvine.

Reminder: If your web and MySQL servers are located in our IRVINE data center, your sites are affected by the current situation. Please check your server locations in your panel at Status > Data Centers:

https://panel.dreamhost.com/index.cgi?tree=status.dc

Please reach us here to receive a final update by email and to raise any other related concerns:

https://panel.dreamhost.com/index.cgi?tree=support.msg

Thank you for your continued patience.

Yes, I will be moving this site to WPEngine.com soon. I have some other sites there… SO much faster and reliable.

I hope it is up soon. I just had an email blast go out to 100s of leads and a potential client could not access my site. That was the way I found out that my site was down =(

Our Chinese immersion learning sites in China are all down as well. Horrible, as it is in the middle of the working day in China at the moment. No students can access the site. Any predictions when the sites will be up again?

Sean, my sites are still down.

How long will it take for the redirection to take place? Site is still inaccessible.

As usual all my websites are on this datacenter.

Thank you guys :-(

What does the fox say?

Today we plan to launch our Corporate Website… But your servers down… GOOD START !!!!

all websites are all down!

Update: All web/MySQL servers have been up, but our border router at Irvine is affecting all inbound/outbound traffic. We are still working on redirecting traffic through our other US-West data center in Los Angeles, which at that point should have all sites resolving again while we continue working on the router at Irvine.

Reminder: If your web and MySQL servers are located in our IRVINE data center, your sites are affected by the current situation. Please check your server locations in your panel at Status > Data Centers:

https://panel.dreamhost.com/index.cgi?tree=status.dc

Please reach us here to receive a final update by email and to raise any other related concerns:

https://panel.dreamhost.com/index.cgi?tree=support.msg

Thank you for your continued patience.

I would recommend everybody file tickets to receive compensation immediately. It’s the least you can do.

My panel says my server is located in Los Angeles but my sites are also not working. Is this affecting users in the LA data center as well besides myself?

This is unbelievable, last time my databases went down it was for 20+ hours. My clients cannot afford this kind of outage nor can I! You would think that there would be some redundancy to keep this kind of stuff from happening. I can see why there are no phone numbers listed for DreamHost, they would be on the phone continuously.

Daniel, I have the same thing, but my sql servers are in Irvine, hence my sites are down.

Dreamhost is my third webhost and best so far. Previous have had even more downtime . . .

I’m taking this opportunity to restart all of my servers, with no loss of connectivity! ;) Serious, though, restart your servers. Me, I’d love to move, but I’m in deep with a majorly complicated server structure and a custom built, highly trafficked e-commerce site. It’s easier to say “move” than it is to actually do so. Plus I’m tired, it’s been a long hard retail season. Thanks heavens it’s January!

you can design services so this will not happen. Servers all go down. Redesign your service guys and quit whining. If your customer is down your service design does not match what you have committed to customer. One of anything will always have downtime. Buy 2…size correctly and loadbalance.

I think everyone here needs to relax. Its not the end of the world. Get a sense of proportion

Weeeeee! My site isn’t hosted at this data center.

To those complaining about DH not having redundancies in place: You may want to keep in mind that DH service is extremely cheap, even for dedicated servers. There are naturally going to be more problems when you’re saving money.

With the exception of a few outages here and there, the DH service is of a higher quality than I would have expected for the price. As far as I’ve noticed, DH does not overload their shared hosting servers like many cheap hosts do, and as a result websites can often load faster.

If you want hosting that rarely if ever goes down, pay more. If you want redundancy, buy it.

This is our 3rd busiest day of the year! we make most of our money in a few days pon our site and thie site has been down now for THREE hours! We want a refund for this month. This is totally unacceptable. def time to move to a more reliable company.

my sites are back on!

My websites are now up…they have been restored http://www.noorkids.com

Woohoooo! Back to work..

So frustrating. I remembered this happened last year at the same time so I started to move my website to a new provider and of course whilst I’m in the middle of it Dreamhost decides to surprise me early and bring all my website down while I’m in the middle of migrating. Thanks a lot Dreamhost. My students now can’t access their course tests as the website on which they are hosted are down because of this. this is the third time in the last 12 months. Extremely unhappy and will be cancelling my hosting with Dreamhost asap.

My Site just came up as well! Thanks so much for all the hard work. I’m sure it has been a stress-filled evening.

9:10 PM, PDT, my sites are back.
Looks like they went down at 7:10 pm.

Oh, and now that I think of it. If you decided to pay for a dedicated server just to escape the networking issues that DH sometimes has, it’s a wonder that you’re even able to manage a dedicated server on your own. Moving to a dedicated server to escape networking issues would only make sense if DH ran dedicated servers in a different data center and/or on a different network, but they don’t. So on behalf of all customers smart enough to realize that all service levels can be affected by networking issues, I’d like to thank you for giving DH the money they need to improve their service for the rest of us. Well done! lol

Thank you !
Just came back from a meeting with a new customer, where I wanted to show the web design portfolio.
Very great when one has to say ” Just right now the server in the US is down.”
Made me look like a stupid amateur !

Really amazing, that there are no redundancies.

Update: We are no longer redirecting traffic through our other US-West (Los Angeles) data center, as we have now restored connectivity through the Irvine border. Sites are coming back online, but we are still working on a resolution.

Reminder: If your web and MySQL servers are located in our IRVINE data center, your sites are affected by the current situation. Please check your server locations in your panel at Status > Data Centers:
https://panel.dreamhost.com/index.cgi?tree=status.dc
Please reach us here to receive a final update by email and to raise any other related concerns:
https://panel.dreamhost.com/index.cgi?tree=support.msg

Thank you for your continued patience as we work to find the root cause!

January 7th, 2014 at 9:36 pmJose Contreras Says:

Thanks for fixing this so quick.

9:28pm PST, just went back down again.

Hey i need my website up!!!!!!!! what the hell is thiss!!

Just went down again.

Should have left it routed thru LA..down again…$$%#@#$%$%%$*$$

The fox says down down down down down-down down-down down… again…

Andrew – same here!

What Ryan said..

yep. hadda glorious 34 seconds to apt-get install pidgin-otr … at least it finished installing before connnection died again. lol. the fun never ends.

It still doesn’t work!
It worked temporary. But now it doesn’t work again! What is your service compared to Amazon’s? I’m thinking to switch!

It’s broken for me still. Can’t chat with support either…

Down Again:(Down Again:(Down Again:(Down Again:(Down Again:(Down Again:(

it is down again guys what is wrong with you

Site still down. I will be “paying” for the uptime. Thought I was already getting my money worth with the slower than average service. Wasn’t informed about the major downtime that I would be experiencing too! I was sold something better with a 100% guaranty. So I give up the slower speeds for a cheaper price. I NEVER agreed to this much downtime!!!!

January 7th, 2014 at 9:43 pmnone of your business Says:

I want my money back.

January 7th, 2014 at 9:44 pmnone of your business Says:

Grant Barnard gets award for longest domain name ever.

Down again, guys.

All my websites have gone down! Please fix ASAP!

We are still not up despite the posted update at 9:09 pm stating that service has been restored.

@ none of your business

Holy mother of God !! www[dot]greenwoodindianapolispainterpainting[dot]com

Yea and dreamhost gets the award for the most downtime ever. LOL I like my award better.

Sites are back up and seem to be staying up now. Hopefully we’re not down again later to repatch what ever got us up for now.

Cheers DH – Please try to keep our servers stable

The websites are up, no ssh access though. And stable is not the word I would use for a site that’s flipping every ten minutes. Especially after Sunday’s interruption. Umm, DH, need some help?

 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled Maintenance for homiemail-sub5 email sub-cluster January 2nd 7:30pm PST (2.5hr) (RESOLVED)
4:12 pm

Scheduled Maintenance for homiemail-sub5 email sub-cluster January 2nd 7:30pm PST (2.5hr) (RESOLVED)

Posted (December 30th, 2013 at 4:12 pm PST) by andrea

On Thursday, January 2nd from 7:30PM-10PM PST, we will be performing maintenance on our “homiemail-sub5″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. This new solution has undergone rigorous testing and we’re confident it will make for a more effective handling of mail on that sub-cluster. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update: January 2nd, 8:50pm PST: We are happy to report that this maintenance is now complete. The homiemail-sub5 email cluster is now being managed by our new software-based load balancer, with a secondary balancer standing by for redundancy. All tests and re-checks came back as expected, and we’re resolving this status post. If you have any questions or concerns, please be sure to contact support with details.

Update: January 2nd, 8:20pm PST: So far so good – the maintenance is going forward as expected, without any perceivable service interruption. We appreciate your continued patience, and we’ll post again when there is status change.
Update: January 2nd, 7:45pm PST: We have initiated the maintenance as scheduled. We will be monitoring the process as well as this status post to make sure everything is going as expected. Please keep referring back here for updates. Thank you for your patience while we conduct this maintenance.

This entry was posted on Monday, December 30th, 2013 at 4:12 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Scheduled Maintenance for homiemail-sub5 email sub-cluster January 2nd 7:30pm PST (2.5hr) (RESOLVED)”

Does this then interveção, problems sending email via outlook and blocking ip (Black List) will be resolved?

Daniel, this maintenance is to resolve access and mail load time issues. If your email sent from DreamHost is being blocked by another provider, please contact support with the details. You can find the support link in the status post above.

I am experiencing slow response times on my website, would this be impacted by this maintenance?

Djack, this maintenance only affects email services. Please submit a support ticket, so we can take a look at you website connection issues.

All the websites on the server “penguins” are down since 15 hours :(

You should make an annoucement about the server “penguins” on this status page…

Why no annoucement about server “penguins” down since 15 hours ?

having problems connecting with POP this morning. timing out

having problems connecting with POP this morning. timing out and getting this error

An operation on the server timed out.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Limited number of servers in US-West (Los Angeles) experiencing connectivity problems (RESOLVED)
3:15 pm

Limited number of servers in US-West (Los Angeles) experiencing connectivity problems (RESOLVED)

Posted (December 28th, 2013 at 3:15 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Los Angeles) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update: December 28th, 4:07 PM PST: The connectivity issues have been resolved. The root cause of the issue was a fuse that needed to be reset on the power strip for that rack. All power has been restored and affected machines are now back online. Should you have any further issue with your sites, please contact support and we will be glad to look into the issue for you!

Update: December 28th, 3:54 PM PST: We have found the root cause of the issue. We are waiting to resolve this post as soon as our internal alerts have dropped, to ensure that this issue doesn’t happen again. Thank you again for your patience as we solve this!

Update: December 28th, 3:26 PM PST: We’ve isolated the issue to a particular rack in our LA datacenter. We are continuing to troubleshoot the issue and will update this post again with more information. The list of affected machines can be found in the comments to this post below. Thank you so much for your patience!

This entry was posted on Saturday, December 28th, 2013 at 3:15 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

7 Responses to “Limited number of servers in US-West (Los Angeles) experiencing connectivity problems (RESOLVED)”

List of affected machines:
bengals
manama
panthers
sarajevo
roughriders
porto-novo
liberty
expos
homie-vserver71
homie-vserver72
homie-vserver74
homie-vserver75
homie-vserver76
homie-vserver77
homie-vserver78
homie-vserver79
homie-vserver80
homie-vserver81
homie-vserver82
homie-vserver83
homie-vserver89
bogota
algiers
thunder
flames
tirane
belmopan
predators
brewers
gaborone
praia
ndjamena
ravens
brumbies
lapaz

My sites are hosted on panthers – they are all down.

@Rob — I am so sorry for the problems! Panthers is definitely one of the affected servers, and we have our admins working on the issue right now.

mine are

web server bogota

MySQL server squirrelfish

and I can not connect to either of them my sites and databases and ftp are down..

I have no complaints though dreamhost rocks :)))))

@Michael — Thank you so much! Bogota is one of the machines affected as well, and we are working hard to get everything back on track!!!

It looks like my VPS is on homie-vserver76, which doesn’t seem to be on the list, but I can’t access it. It looks to have went down around 3pm PST which correlates to this time here. Can you please confirm if homie-vserver76 is part of this outage?

@Jeff — I’m so sorry for the confusion! I had the list of VPS machines in a weird order — I’ve gone ahead and put the in numerical order, and homie-vserver76 is part of this outage.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issue in one row of US-West (Irvine) Data Center (RESOLVED)
8:25 pm

Connectivity Issue in one row of US-West (Irvine) Data Center (RESOLVED)

Posted (December 18th, 2013 at 8:25 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Irvine) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update December 18th, 8:56 pm PST: We have mitigated the abusive traffic and there should no longer be issues on any server in Irvine. If you are experiencing any problems, please contact support and let us know! Thank you for your patience!


Update December 18th, 8:36 pm PST: We are still working on blocking any possibly abusive traffic, so we should be seeing better response times very shortly. We are observing notable improvement, so hold on and thank you for your patience!

Update December 18th,  8:25 pm PST: The connectivity issue is due to a massive amount of  inbound traffic to the Irvine data center. This traffic is only affecting a specific row in our data center, so it is limited to a small group of customers. We are working on blocking the traffic and restoring connectivity to the customers who are having problems reaching their sites. Thank you for your patience!

This entry was posted on Wednesday, December 18th, 2013 at 8:25 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects – Trouble With Large Objects (RESOLVED)
10:12 pm

DreamObjects – Trouble With Large Objects (RESOLVED)

Posted (December 17th, 2013 at 10:12 pm PST) by Steven

Following today’s update of Ceph, the software that powers DreamObjects, we have been seeing issues with large objects (> 500k in size). Troubleshooting began during scheduled maintenance. Our DreamObjects engineers began looking into the problem, identified it, and a fix is forthcoming. We apologize for any inconvenience this may cause and thank you for your patience! Please check back for updates as we will be posting them as soon as they become available.

This applies to DreamObjects only, no other services are affected. If you have any questions or need anything else please be sure to Contact Support.

Update December 18th, 12:28am PDT The fix has been implemented and calling this resolved! We have not found any other issues with our testing but please do not hesitate to Contact Us. Thanks again for your patience!

Update December 18th, 12:15am PDT The fix is looking good and we are preparing to roll it out. We are, however, being cautious and continue to monitor/test for any issues.

Update December 17th, 11:49pm PDT Testing continues at this moment. We apologize for the delay and look forward to resolving this soon.

Update December 17th, 11:06pm PDT This is just a quick update to let you know we are currently testing the fix and preparing to carefully roll it out.

Update December 17th, 10:22pm PDT Some additional details about the situation: only about 4% of all incoming requests, be it bucket creation or file access, have been failing. We are continuing to investigate the cause, and we apologize for any inconvenience this is causing.

This entry was posted on Tuesday, December 17th, 2013 at 10:12 pm and is filed under DHO. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.8 Major Release Now Available
10:31 am

WordPress 3.8 Major Release Now Available

Posted (December 12th, 2013 at 10:31 am PST) by Mika E.

First it was WordPress 3.6 in August and then WordPress 3.7 in October, and now it’s December and surprise, it’s WordPress 3.8!

The newest version of WordPress (3.8 aka “Parker” after Charlie Parker), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

While 3.7 was a sneaky release, trust me, you’ll notice 3.8. The dashboard had an overhaul, with new colors for everyone’s preferences (no more just blue and grey!), and the whole design is modern and responsive. That means all your smart phones and tablets will be able to use WordPress with even greater ease than ever before! The theme dashboard got a facelift too, as did widgets, so you’ll find changing your site’s design to be far easier.

Now, we understand the update fatigue. A new major release every 2 months is tough (especially on us). We’ve got your back, though! Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.8 in the DreamHost forums. Unlike previous releases, there is not (at this point) any plan to make an official ‘issue tracking’ thread on WordPress.org forums, but please do go there to report any problems you have with specific plugins and themes. Every single plugin developer was contracted, prior to release, to remind them to test, so we’re hoping that worked. It was a very large email list.

Remember, only minor releases (like 3.8 to 3.8.1) will now be automatically applied by WordPress. We’ll still run our One-Click updates for everyone, just in case the automatic tool doesn’t work (or is disabled). But you’d never disable that, right?

This entry was posted on Thursday, December 12th, 2013 at 10:31 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “WordPress 3.8 Major Release Now Available”

So if you don’t have a smart phone or tablet and don’t care about cosmetic design tweaks, you can skip this release, right?
Was scanning this post and Mullenweg’s for any information about security issues, which are the user’s main concern, release-wise. Didn’t see anything.
Also, calling the release “Charlie Parker” is pretentious. The designers are comparing themselves to genius jazzmen (minus the heroin addiction, money problems, etc.).
You said it all with the phrase “update fatigue.”

It seems like you know what you are talking about. If you want a $50 discoun on hosting with DreamHost then just use the promo code SAVEHUGE50 at checkout.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail Upgrade for Homiemail-Sub1, Wednesday December 11th 6pm – 10pm(POSTPONED)
6:03 pm

Mail Upgrade for Homiemail-Sub1, Wednesday December 11th 6pm – 10pm(POSTPONED)

Posted (December 10th, 2013 at 6:03 pm PST) by Ralph

On Wednesday, December 11th from 6PM-10PM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update December 11th 3:00PM PDT: We have unfortunately decided that it would be best to postpone this maintenance once more. As we near any maintenance, we are constantly evaluating our plan and monitoring service to make absolutely sure that our maintenance will not only run as smoothly as possible to prevent any further service interruptions, but also address and correct the problems we’re experiencing at this time. We will be revisiting this maintenance in the near future, once we are confident that we can complete it successfully.

This entry was posted on Tuesday, December 10th, 2013 at 6:03 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Scheduled Maintenance – December 17th, 9am PDT – 19th, 5pm PDT (RESOLVED)
4:52 pm

DreamObjects Scheduled Maintenance – December 17th, 9am PDT – 19th, 5pm PDT (RESOLVED)

Posted (December 6th, 2013 at 4:52 pm PST) by andrea

Our Systems Engineering team will be performing maintenance on our DreamObjects service from Tuesday December 17th through Thursday December 19th. During this maintenance period, you may intermittently experience degraded performance. The purpose of this maintenance is to implement key upgrades to Ceph, the underlying storage system which powers DreamObjects. As a result of the upgrades, we’ll be able to provide some exciting new features, including CORS support and improved multi-part uploads. While we anticipate increased latency when storing or retrieving objects, please rest assured that your data is safe and will not be affected.

If you have any questions or concerns about the upcoming maintenance, please contact our support team.

Update December 20th, 11:40am PDT Just a quick note to let you know that as previously mentioned, the upgrade of this service has been successful. We resolved the post yesterday, but please let us know by href=”https://panel.dreamhost.com/index.cgi?tree=support.msg&” target=”_blank”>contacting our support team if there is anything else we can do for you.

Update December 18th, 12:30pm PDT Just a quick update to let you know that we have completed the upgrade that was the main objective of this maintenance. We are currently testing and double checking the processes to make sure everything works as expected. We appreciate your continued patience.

This entry was posted on Friday, December 6th, 2013 at 4:52 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Stability Issues Impacting a Small Number of Hosting Servers in US-West (Los Angeles) (RESOLVED)
1:52 pm

Stability Issues Impacting a Small Number of Hosting Servers in US-West (Los Angeles) (RESOLVED)

Posted (December 6th, 2013 at 1:52 pm PST) by ScottK

Memory issues are causing a subset of servers on our US-West (Los Angeles) data center to become unresponsive. We are seeing similar behavior in a couple of other data centers as well but on a much smaller scale. We are working with our network, admin and security teams to determine the cause and fix for this specific issue. We will provide additional information and servers affected when available.

Update December 5th, 14:48 PDT – The server stability issue has been resolved. Our team is continuing to investigate the matter to ensure that it does not occur again. Thank you again for your patience.

Update December 5th, 14:13 PDT – Improvements to server stability have been made and alerts are continually dropping. We are still investigating the root cause and will update this status post with more information as soon as possible.

Update December 5th, 14:06 PDT – We are currently working on all affected servers individually and recovering services on them. We have our entire team addressing this issue.  Thank you for your patience.


This entry was posted on Friday, December 6th, 2013 at 1:52 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Stability Issues Impacting a Small Number of Hosting Servers in US-West (Los Angeles) (RESOLVED)”

December 6th, 2013 at 2:04 pmMichael Palmer Says:

Both my wife and I (for our email on dreamhost-hosted domains) have been having intermittent trouble for past 1+ week sending email. Very similar to symptoms in this other issue:

http://www.dreamhoststatus.com/2013/12/05/stability-issues-impacting-a-small-number-of-hosting-servers/

A workaround: We experience the problem when sending email from Thunderbird email client on OS X. However, if I save to Draft folder from thunderbird, then go to my iphone, I can pull up the draft and send it from the phone. Don’t ask me why this works, it’s pretty strange. (Maybe the phone client and thunderbird interact slightly differently with the server, no idea.)

I do not believe the problem is related to email settings on either device – the problem arose for both my wife and I independently without changing the settings in Thunderbird.

The error message I get on thunderbird is “Send Message Error Sending of Message failed. The message could not be send because the connection to SMTP server mail. timed out. Try again or contact your network administrator.” (I removed my domain from the message.)

@Michael try setting the SMTP port to 587 in Thunderbird.

@Michael – we are working on mail issues as aggressively as we can. The mail issues experienced are not related to the stability issues experienced yesterday and today. Mail issues are intermittent and in some cases is going to require a hardware refresh. Making mail a stable reliable service is one of our major goals and our team is pushing towards it. Apologies for the issues there. We will continue to work and improve our mail service daily.

Also having troubles with mail (pop3-110, smtp-587) since +2 weeks, no word from technical support.
I also get my isp to change routers.
This is upssetting…

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Stability Issues Impacting a Small Number of Hosting Servers (RESOLVED)
11:39 am

Stability Issues Impacting a Small Number of Hosting Servers (RESOLVED)

Posted (December 5th, 2013 at 11:39 am PST) by Justin@DH

Memory issues are causing a subset of servers on our US-West data center to become unresponsive. We are seeing similar behavior in a couple of other data centers as well but on a much smaller scale. We are working with our network, admin and security teams to determine the cause and fix for this specific issue.  We will provide additional information and servers affected when available.

Update December 5th, 12:03pm PDT: Improvements to server stability have been made and we are seeing less issues that could affect customers. We are continuing our efforts to resolve the remaining issues.

Update December 5th, 12:40pm PDT: Our monitoring system alerts have dropped significantly at this time. We aren’t resolving this post yet to sure the root case is found and fixed. When we have a final resolution or other chance of situation we will update this post.

Update December 5th, 1:03pm PDT: We are examining a large amount of anomalous network traffic that we think is related. We will post further details as we have them. We are resolving the post but are continuing to investigate and monitor the situation closely.

This entry was posted on Thursday, December 5th, 2013 at 11:39 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

11 Responses to “Stability Issues Impacting a Small Number of Hosting Servers (RESOLVED)”

When can we expect the sites to be back up?

Why have you still not posted any status update here about the ongoing issues that you’re having with homiemail-sub5? You acknowledge in tickets that it’s been ongoing for over two weeks now, yet nothing gets fixed, and customers are just getting automated messages in response to tickets.

Seconded on ISPMGT’s comment. homiemail-sub5 was apparently under SPAM attack yesterday, and today we’re continuing to have problems. Is that related to this or a separate issue??

When will you be fixing homiemail-sub5? We have been experiencing problems for 2 weeks and have not been able to send emails for 2 days. Got an automated message that it was fixed from yesterday but today having the same problem again.

Agreed… homiemail-sub5… I am running a business and I cannot send any emails. It was bad about a week ago; got better; and then started up again yesterday. I also have my fax set up to email faxes to me and it is giving me errors as it can’t read the email server. This is very inconvenient! I have to run my business and can’t reply to my clients at all! I have had no responses to my last 3 tickets and see nothing posted here on the status page.

iad1-vshost52 is having problems as well. I’m getting host server not responding errors.

same here, intermittant send issues for 2 1/2 weeks. when pushed tech says load balancing is not stable. but no details of what is broken and how they are working to fix it. nothing in status except the Nov 25 note that is marked resolved.

Is this a reoccurring problem for anyone else? We were first told that it was a heavy server usage issue, bot issue, ddos issue, powercycle issue, bandwidth issue. I see that ISPMGT Says:
Why have you still not posted any status update here about the ongoing issues that you’re having with homiemail-sub5? You acknowledge in tickets that it’s been ongoing for over two weeks now, yet nothing gets fixed, and customers are just getting automated messages in response to tickets.

Jeremy Says:
Seconded on ISPMGT’s comment. homiemail-sub5 was apparently under SPAM attack yesterday, and today we’re continuing to have problems. Is that related to this or a separate issue??

Linda Says:
When will you be fixing homiemail-sub5? We have been experiencing problems for 2 weeks and have not been able to send emails for 2 days. Got an automated message that it was fixed from yesterday but today having the same problem again.

These all seem to be similar situations to our problem with different excuses. Our problem has been going on since Oct 6th with just a band aid and no real lasting solution.

We we get a real fix today?

This is the second time in a week this has happened and service still isn’t restored. Can we get an update. I have customers complaining around the clock.

dreamhost,

PLEASE publicly acknowledge on this status site the on going issues with homiemail-sub5. i have been experiencing the same issues as mentioned above by numerous people and have been given the same responses… yet the outages keep happening day after day.

please stop with the canned responses and actually fix the issue with our email.

I can send emails but when someone replies back it doesn’t go through??Would this be my problem

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects currently experiencing service interruption (RESOLVED)
11:20 am

DreamObjects currently experiencing service interruption (RESOLVED)

Posted (December 5th, 2013 at 11:20 am PST) by Ralph

DreamObjects is currently experiencing a service interruption. We first noticed the situation around 10:53am PST. Our team is currently investigating this unexpected downtime. More details to come. No other services are affected by this downtime, only DreamObjects. We will update this status post as we find out more about this situation. At this time we do not currently have an ETA for return of service. Apologies for the interruption, we will do our best to get the service running as soon as possible. Your patience is greatly appreciated.

Update December 5th 11:20am PST: DreamObjects is now working correctly. We have determined the load balancer to be the cause of the issue. Our team will continue investigation to make sure that this type of problem can not come up again. Apologies for the unexpected issues.

This entry was posted on Thursday, December 5th, 2013 at 11:20 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issue in US-West (Irvine) Data Center (RESOLVED)
1:34 pm

Connectivity Issue in US-West (Irvine) Data Center (RESOLVED)

Posted (November 25th, 2013 at 1:34 pm PST) by Sean

We are currently experiencing a connectivity issue in our US-West (Irvine) data center. We’re investigating the cause and are working towards a resolution. We apologize for the inconvenience this is causing, and will post updates as the work progresses. Web servers, mysql and email may be affected by this issue. Thanks for your patience!

Update November 25th 1:55PM PDT: All hosts are up and responsive, we’re in the clear. If you’re experiencing any issues, please contact support! Thanks again for your patience.

Update November 25th 1:39PM PDT: The new switch is now responding, and hosts are accessible again. We will leave this post open until we’re 100% sure things are in the clear. Thanks for your patience!

Update November 25th 1:36PM PDT: We have identified the issue to be caused by a single switch, which we’ve replaced, and we’re currently copying the config from the old switch.

This entry was posted on Monday, November 25th, 2013 at 1:34 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

12 Responses to “Connectivity Issue in US-West (Irvine) Data Center (RESOLVED)”

Thanks for the update Sean. All of my sites are down, so I hope we are back online soon!

Error establishing a database connection

Thanks, Lawrence. We’re still clearing everything up, and I’ll be sure to check on your site momentarily!

Unable to access any database. It’s been like this for hours now.

@Lawrence: Your database user password is incorrect in your wp-config.php file. Update that and your site will work again.

Not true, my site is still down
“Update 1:39p : The new switch is now responding, and hosts are accessible again. We will leave this post open until we’re 100% sure things are in the clear. Thanks for your patience!”

thanks oscar! how can I edit that file? ftp?

Marcos, it sounds like your issue may be unrelated. Please reach support from your panel at Support > Contact Support, or via live chat! Give us the details and we’ll get right on it.

@lawrence: Yes, you can use FTP to download, edit and then re-upload the file or you can edit it directly on the server using ssh. I fixed this for you and the site is now back up and resolving.

500 Internal Server Error in a Prestashop store.

My site has being showing this error all day is this related?

Rodrigo, no, that would not be related. I will have a look and send you an email!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail upgrade for homiemail-sub1 cluster, Dec 2nd 1AM-2AM (1 hour maintenance) (RESOLVED)
2:05 pm

Mail upgrade for homiemail-sub1 cluster, Dec 2nd 1AM-2AM (1 hour maintenance) (RESOLVED)

Posted (November 21st, 2013 at 2:05 pm PST) by Seohee

On Monday, December 2nd from 1AM-2AM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers, which should help alleviate recent issues. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update December 1st 1:30PM PDT: Unfortunately, we ran in to some issues with the migration and we will not be ready to perform this upgrade at the scheduled time. We will be rescheduling it, but don’t have a date or time for it yet. We will create a new status post announcement once we have scheduled a new timeframe. We’re sorry for any inconvenience this causes you.

This entry was posted on Thursday, November 21st, 2013 at 2:05 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail upgrade for homiemail-sub1 cluster, Nov 25th 1AM-2AM (1 hour maintenance) POSTPONED
3:22 pm

Mail upgrade for homiemail-sub1 cluster, Nov 25th 1AM-2AM (1 hour maintenance) POSTPONED

Posted (November 20th, 2013 at 3:22 pm PST) by Brandon

On Monday, November 25th from 1AM-2AM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers, which should help alleviate recent issues. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update Nov. 21st 2:45pm PST: This maintenance has been temporarily postponed to December 2nd. New status post has been created. We have determined that our plan for the upgrade needed some further changes to ensure that the upgrade will go as smoothly as possible and provide a superior level of service once complete.

This entry was posted on Wednesday, November 20th, 2013 at 3:22 pm and is filed under Announcement, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail Issues with homiemail-sub3 and homiemail-sub4 (RESOLVED)
7:15 am

Mail Issues with homiemail-sub3 and homiemail-sub4 (RESOLVED)

Posted (November 20th, 2013 at 7:15 am PST) by Justin@DH

We are experiencing issues with a specific file server that has lost connectivity to the network.  This is causing service degradation for servers in the listed mail clusters which has the potential to disrupt email access (both receiving and sending).  Members of every team are working on the issue at this time and hope to have it resolved soon. We will provide more information as it becomes available.

If you are unsure if you are impacted by this issue please check the account status drop-down on the top right of the account management panel . Your email cluster is listed at the bottom.

Update November 20th 7:30am PDT:Connectivity to the file server has been restored by our teams.  The mail servers using that machine will need to catch up on all the data they attempting to write to it and so may be slower than normal for a short period of time. Just to be safe, we are monitoring these servers to make sure they recover (we anticipate normal mail service will be restored shortly once everything is caught up).

Update November 20th 7:55am PDT:Connectivity is still looking fine. The mail machines are still recovering from the file server issue, but are nearly back to normal. There may be some delays on receiving messages until all the queues on the mail machines have caught up. We are continuing to monitor these servers to make sure they recover (we anticipate normal mail service will be restored shortly).

This entry was posted on Wednesday, November 20th, 2013 at 7:15 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

23 Responses to “Mail Issues with homiemail-sub3 and homiemail-sub4 (RESOLVED)”

Any estimated ETA on a fix time?

Not able to send/receive mails. Can you please update with ETA?

Like everyone else, what is the ETA?

@James & @Zaheer & @Corinne: We have staff on site already, and debugging is underway. We hope to have it fixed very soon.

There is nothing Thankful going on here. ETA???????

Another outage … ? Have an ETA to get back to business please?

@ALL: The filer server connectivity is now restored. I’ll update the post shortly. The mail servers may have higher loads than usual and so be a bit slower, but will recover fully soon.

Did homiemail open an ancient Egyptian tomb or something? It seems to be cursed.

I can connect now but the mails I’m sending to myself as a test aren’t being delivered. I guess it may be because there’s a queue?

@Ab: Yes, mail that was trying to be delivered during this issue will be queued up but should arrive shortly.

We cannot sending receiving mail

We are on homiemail-sub5 and can receive email but not send any email. Is this part of the same problem or another problem?

this seems to be a pattern…. can’t log into webmail. Last chance before I jump ship and go back to go daddy…. more expensive, but it works.

We can still not receive any emails…:( Please HELP!!!

I’m on homiemail-sub5 and cannot retrieve or send email. Is this the same issue?

In Argentina we have have “OK” in pop and smtp servers, but:

1) Emails are not effectively dispatcher.
2) in other cases are rejected.
3) Do not have access to IMAP servers.

We hope to resolve the case.

not resolved dreamhost

@ALL: Mail server loads are still elevated but dropping. We hope to have all loads and services back to normal within 15 minutes.

Actually, I can access everything via webmail, but not from iPhone or Thunderbird on PC. Sounds like an IMAP problem.

I second that. NOT resolved. Imap emails not working!

I’m on sub4:
When attempting to login via POP3 as of 9:06AM CST:
-ERR [IN-USE] Internal login failure. Refer to server log for more information.

Seems OK now though.

I got only one of the several test emails I sent – earlier tests and other emails people told me were sent have *not* come to webmail or IMAP – so either the queue is not chronological (which I hope id the case) or some of my incoming mail was lost without the sender getting a bounce – would be Really A Bad Thing.

@Brian: They aren’t chronological so it should arrive in time.

@All: There are only a few machines in the load balance cluster still with higher loads, so all should be back to normal now. There can still be some delays with email but they should be on the shorter side.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance Affecting All Data Centers Nov. 19th 10pm – Nov. 20th 2am PST(4 hours) (RESOLVED)
2:19 pm

Network Maintenance Affecting All Data Centers Nov. 19th 10pm – Nov. 20th 2am PST(4 hours) (RESOLVED)

Posted (November 15th, 2013 at 2:19 pm PST) by Ralph

On Tuesday, November 19th, starting at 10pm PST we will be performing network maintenance in our three data centers:

US-West (Los Angeles), US-West (Irvine) and US-East (Virginia)

This maintenance will improve communication between our three data centers.

The changes will require propagation across the internet via the BGP routing protocol. As a result, your service may appear to be down during the maintenance for a short period that should not exceed 5 minutes. Service interruption will largely depend on your ISP and how quickly the changes register within their network. To help identify any problems, we will have our staff monitoring the changes from multiple locations and ISPs.

We will be performing this maintenance in three stages with one for each data center. We will start with our US-West (Los Angeles), followed by US-West (Irvine) and finish up with US-East (Virginia). We will provide updates as we move through the maintenance.

To see which data center your site is hosted in, please look at Status->Data Centers section in the panel.

Comments will be opened when the maintenance starts so that we can assist you with any issues you may be noticing.

If you have any questions about the upcoming maintenance, please contact our support team.

Update November 20th 1:50am PDT: That last update went perfectly! We are now in finished, and can close out this maintenance window. We appreciate everyone’s patience and are happy to report no issues. If you see your site having problems, please contact our support team and let them know!

Update November 20th 1:40am PDT: We’re nearing completion, but this last step could cause some downtime for our US-East (Virginia) datacenter. We will update when this step is complete. We appreciate your patience!

Update November 20th 1:05am PDT: Ok, we’re now starting our final section, the US-East (Virginia) datacenter. We don’t anticipate problems on this, however the possibility exists. Please let us know via the comments if you see any problems.

Update November 20th 12:50am PDT: We are currently finishing up our Our US-West (Irvine) datacenter here soon, and will be moving on to our US-East (Virginia) datacenter. We’ll update when we move on to that area.

Update November 20th 12:25am PDT: The Our US-West (Irvine) datacenter is large, so we’re plugging away. So far we have seen no issues related to this maintenance window. If you see any problems, please let us know immediately via the comments below.

Update November 19th 11:35pm PDT: We are still updating subsets of our US-West (Irvine) datacenter, everything is going according to plan.

Update November 19th 11:00pm PDT: Our US-West (Los Angeles) datacenter is all moved over, and now we’re ready to start on the US-West (Irvine) datacenter. As before, if you notice downtime or problems, please mention that in the comments below, and we’ll address them ASAP.

Update November 19th 10:30pm PDT: We are now ready to begin and will be updating networks in our US-West (Los Angeles) data center. So far we have not seen any issues, however the possibility of downtime is there. If you see problems, please let us know in the comments section below, and we’ll address them for you!

Update November 19th 9:57pm PDT: We are here and are gathering our troops to get this maintenance started. We’ll update when we are ready to begin!

This entry was posted on Friday, November 15th, 2013 at 2:19 pm and is filed under Data Center Maintenance, General Outages, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

7 Responses to “Network Maintenance Affecting All Data Centers Nov. 19th 10pm – Nov. 20th 2am PST(4 hours) (RESOLVED)”

My website is EXTREMELY slow today.

I submitted a support ticket about eight hours ago, but have received no reply yet.

Is my issue related to this maintenance?

Looks like you already moved to US-East (Virginia) datacenter? My website is down.

@Steven, I’m having a tech check on it for you, your issue is unrelated to this maintenance.

@Russ, I tested your site, and it seems to be working, are you still seeing problems?

Hi Brian,

Yes still the same site is offline, currently is loading using Cloud Flare Always Online option.

@Russ, ok, found the problem, unrelated to the maintenance. We’re fixing that right up, however!

@Russ – All fixed! You should see the site loading directly now instead of through CloudFlare.

Hi Brian, thanks you for doing it, much appreciated.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Future PHP upgrades – your input is wanted!
5:26 pm

Future PHP upgrades – your input is wanted!

Posted (November 14th, 2013 at 5:26 pm PST) by andrea

As you may have noticed, we now offer three version of PHP on our web servers. 5.2 is really on its way out and isn’t recommended any more. You can read about it here.  The version most likely to work with common web applications  is 5.3, but we did make 5.4 available for those of you who wanted to move ahead with that. Please note that with any version of PHP, you may be noticing incremental upgrades and patches from time to time. If your domain is set to run 5.4, these upgrades may or may not affect you. If they do affect you, you will have to either remove/upgrade the affected application or downgrade your php version for that domain in the DreamHost panel to 5.3. (edit your domain and scroll down to Web Options) That said…

Due to some incompatibility concerns that arose since the last time we upgraded to the latest PHP version, we are starting a conversation with you, the customer, to gain better insight and to provide a forum for you to contribute your findings, ideas and questions and concerns.

We welcome you to share your thoughts over in the PHP Versions: Upgrades, Observations, and Troubleshooting thread.

We will do our best to post all version upgrades (incremental or full), and will be monitoring the thread for your feedback. We would like to hear from you: what did you notice, did it affect your site, what exactly was affected, etc. We are hoping this to become not just an information exchange among our customers, but that we can also learn from it to better prepare for our upcoming upgrades. We do the upgrades to continue protecting the servers your sites are hosted on, but we understand that there is always room for improvement!

This entry was posted on Thursday, November 14th, 2013 at 5:26 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress Theme Vulnerability: Themify
3:21 pm

WordPress Theme Vulnerability: Themify

Posted (November 13th, 2013 at 3:21 pm PST) by Mika E.

This morning, Themify announced that there was a security vulnerability in their theme. This vulnerability was caused, in part, by their own upgrade process which did not delete a vulnerable file. As a result of this announcement this morning, we, DreamHost, blocked the file from being able to be used to hack into sites.

It’s important to note that the fix we applied only works if you have Mod Security enabled. Most of you should, it’s default in our Panel. But if you, for some reason, turned it off, we really think you oughta go back and make sure that “Extra Web Security?” is checked in your domain’s edit page.

While this means your sites are safe, if you’re using Themify we strongly suggest you reinstall their theme template manually. You’ll want to download a fresh version of their framework, delete the themify folder (NOT your child themes, don’t panic, this won’t break your site’s layout), and upload the new one. You can’t just overlay the files, sadly, you really have to delete the old folder for this to be fully patched.

For more information, please read the Themify Blog — URGENT: Vulnerability Found in Themify Framework, Please Read.

Please contact our tech support team if you have any questions regarding this announcement.

This entry was posted on Wednesday, November 13th, 2013 at 3:21 pm and is filed under Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “WordPress Theme Vulnerability: Themify”

weofijaweofjowjfjifawoefajwoefapwfeowpeafjipa

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Service change notifications delayed (RESOLVED)
2:44 pm

Service change notifications delayed (RESOLVED)

Posted (November 8th, 2013 at 2:44 pm PST) by andrea

Due to a hardware failure, service notifications generated by our DreamHost panel are temporarily delayed. Please note, the service changes you make in the panel are are going through, it’s just the confirmation messages of those changes, like creating an email address, adding a domain, changing an FTP user, essentially all the service management you can do on the DreamHost panel, that are taking a bit longer than usual.
We are working on getting this situation remedied, and appreciate your patience in the meantime. Please check back for updates.

Update November 8th 3:05pm PDT: We are happy to report that the service change notifications are back on track. Please accept our apologies for any inconvenience their delay may have caused.

This entry was posted on Friday, November 8th, 2013 at 2:44 pm and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Central database maintenance – Friday November 8, 3:45am PDT (RESOLVED)
12:28 am

Central database maintenance – Friday November 8, 3:45am PDT (RESOLVED)

Posted (November 8th, 2013 at 12:28 am PST) by chalise

At 3:45AM PDT, we will be performing maintenance on our central database. The DreamHost panel (https://panel.dreamhost.c0m) will be inaccessible during this time, but will not cause any downtime for your sites, email, databases, etc. This only affects the panel itself. We expect the maintenance to be done by no later than 4:45am PDT.

We apologize for the lack of advanced warning and will update this post again once the maintenance begins. Thanks for your understanding!

UPDATE Nov 8, 2013; 4:38am PDT: Maintenance has now completed and the panel is accessible once again! Thanks once again for your patience and understanding!

UPDATE Nov 8, 2013; 3:47am PDT: We are now beginning maintenance on our central database. As noted above, this will not cause any issues with your sites, and only affects the panel, which will be temporarily inaccessible during this period. We will update again once the maintenance has completed. Thanks for your patience!

UPDATE Nov 8, 2013; 3:38am PDT: We’re going to go ahead with the maintenance shortly, apologies for the delay. As promised, we will update this again when the maintenance begins!

UPDATE Nov 8, 2013; 3:09am PDT: It looks like we’re going to go ahead and reschedule this for another time. We will update this post again with further info when the new time is announced!

 

This entry was posted on Friday, November 8th, 2013 at 12:28 am and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – Wednesday November 6th 10am PDT (8hrs) (POSTPONED)
4:09 pm

DreamObjects Maintenance – Wednesday November 6th 10am PDT (8hrs) (POSTPONED)

Posted (November 1st, 2013 at 4:09 pm PST) by andrea

This maintenance is the second part of DreamObjects Maintenance – Wednesday October 30th 10am PDT

As we were not able to complete the expected improvements, we need to continue and complete this work on Wednesday, November 6th from 10 AM PST to 6 PM PST. We do not anticipate service interruption (the issues that came up previously have been completely resolved) but we provide the notice and window so that you are aware of what’s going on.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update November 6th 6:09pm PST: Unfortunately, we ran into some difficulties while trying to complete the maintenance. They weren’t at the point of affecting your service availability, but we didn’t want to take any chances. We will be re-evaluating our process and will re-attempt the maintenance at a future date. We will add a status post as soon as we know the date and you will also receive an email if you have DreamObjects service with us. Sorry about any inconvenience this may have caused.

Update November 6th 3:37pm PST: The maintenance has been progressing as scheduled. We had a minor bump in the road a few minutes ago, so you may have seen some errors, but it has since been resolved. We appreciate your patience while we are moving towards completing this process.

Update November 6th 12:50pm PST: We had a slight hiccup earlier that might have made your access to your objects a bit slower than usual. That has been fixed, and we’re continuing with the maintenance on schedule.

Update November 6th 11:27am PST: I’m happy to report that over an hour into the maintenance, everything is going as expected. Please keep an eye on this post for upcoming updates, but as before, we don’t expect any service interruptions.

Update November 6th 10:17am PST: This update is to let you know that we started the maintenance. We are not anticipating customer-facing service interruption, regardless, we are closely monitoring all steps needed to be taken, and will keep you updated throughout the process.

This entry was posted on Friday, November 1st, 2013 at 4:09 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » UPS Maintenance: US-East (Virginia) Datacenter (RESOLVED)
4:12 am

UPS Maintenance: US-East (Virginia) Datacenter (RESOLVED)

Posted (October 31st, 2013 at 4:12 am PST) by chalise

We are beginning our scheduled annual maintenance on our UPSes in our US-East (Virginia) Datacenter.

Revised maintenance windows:
November 1st, 5 AM – 7 AM PST
November 1st, 7 AM – 2 PM PST
November 4th, 4 AM – 2 PM PST
November 5th, 4 AM – 2 PM PST

This is not expected to cause any downtime, however, we will keep this post updated with major developments as the maintenance progresses.

Update Nov 5 2:47pm PST: The final phase of our maintenance is complete. As expected this did not have any negative impact on customer sites.

Update Nov 5 4:03am PST: We are now beginning the final phase of our maintenance. As with the last maintenance windows, we do not expect this to cause any downtime, and will update when it has completed, or if we do experience any issues.

Update Nov 4 1:07pm PST: The maintenance for today is now complete. Again no problems were encountered. The next maintenance window begins on Tuesday morning at 4am PST.

Update Nov 1 2:23pm PST: The second round is now complete. No problems were encountered. The next maintenance window begins on Monday morning at 4am PST.

Update Nov 1 7:11am PST: The first round of maintenance is now complete. Everything appears to have gone smoothly. The second round is starting up. Again this phase should not cause any downtime. We will update again if there are any issues or once this phase is complete. Thanks!

Update Nov 1 5:00am PST: Today’s first round of maintenance is beginning now! As previously noted, since this should not cause any downtime, we will only be posting status updates as each window completes and finishes, unless there are any issues along the way.

Update Nov 1; 12:45am PST: Apologies for the lack of updates! Maintenance was not finished within the expected window, so the schedule has been updated above. Since this should not cause any downtime, we will only be posting status updates as each window completes and finishes, unless there are any issues along the way.

Update Oct 31; 6:10am PST: Everything is continuing as expected, with no issues to report! We will update again in a few hours, or sooner if needed.

This entry was posted on Thursday, October 31st, 2013 at 4:12 am and is filed under Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “UPS Maintenance: US-East (Virginia) Datacenter (RESOLVED)”

I host numerous sites through y’all, and all of them are down this morning in New Mexico, USA. Could your maintenance be the cause of this? (I receive 500 errors when trying to access them.)

I’m getting internal server error/error 500 as well. Cached pages are popping up, but anything dynamic is throwing internal server errors. Can’t log into wordpress, can’t perform searches, if a page doesn’t have a cached copy it’s throwing up errors.

@Chris – No, as noted in the updates, thankfully, none of the maintenance has caused any downtime. It looks like you submitted a ticket, and have also received a reply. If you have any further questions, go ahead and send a reply to the response you received!

@Gamerfish – It looks like we replied to your open ticket, go ahead and reply back to any of those if you have any further questions!

Thanks!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Monday October 26th 8pm – October 27th 10am PDT
7:10 pm

Live Chat Unavailable Monday October 26th 8pm – October 27th 10am PDT

Posted (October 25th, 2013 at 7:10 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Saturday, October 26th, from 8pm to 10am PDT (the following morning). We will be involving our support staff in our annual Halloween party, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 10am PDT the next morning, Oct 27th. We thank you for your patience and understanding.

This entry was posted on Friday, October 25th, 2013 at 7:10 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – Wednesday October 30th 10am PDT (6hr)(POSTPONED)
1:22 pm

DreamObjects Maintenance – Wednesday October 30th 10am PDT (6hr)(POSTPONED)

Posted (October 25th, 2013 at 1:22 pm PST) by andrea

On Wednesday, October 30th from 10AM-6PM PDT, we will be performing maintenance on the network that DreamObjects uses. The maintenance work includes adjusting our address scheme to a more resilient design, so that we can reduce potential points of failure. We are not expecting any customer-facing downtime during the maintenance, regardless, our team will be closely monitoring the process.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

==================
About DreamObjects
DreamObjects is DreamHost’s object storage service. Object storage is like a giant hard drive on the internet that can never fill up. It’s designed to store static content, so it’s great for images, videos, application data, logs, etc. There’s even a WordPress plug-in to backup your site to DreamObjects! Store it reliably and cost-effectively with DreamObjects. Learn more at DreamObjects.

Update October 30th 4:00pm PDT: We are confident that we are no longer experiencing any issues caused by today’s maintenance. We will need to postpone the remainder of this maintenance to next week. Our confidence is high that we can perform next week’s maintenance with largely unnoticed impact. We will however conduct the maintenance in a window, in case anything unexpected should occur. We sincerely apologize for any inconveniences this has caused and we thank you for your continued patience. When we are confident to move forward next week, we will be providing direct email contact, as well as another status post, to detail our next attempt.

Update October 30th 2:26pm PDT: We’re happy to report that we are in the clear and have been for a little while. We would like to apologize again for the service interruption. We’re moving forward with the maintenance. Please keep an eye out for further updates.

Update October 30th 1:30pm PDT: We sincerely apologize for the service impacts being experience now. With any maintenance there is a risk of interruption. We have done similar maintenances with no issues, so our initial estimates were incorrect. We are still working towards 100% resolution, however we do realize that some customers are still affected by this maintenance. Any new user creations that have failed will be fixed once we get this all ironed out. Thank you for your patience.

Update October 30th 11:35am PDT: The maintenance started as scheduled. We ran into a slight issue which we are working on eliminating right now. It shouldn’t be causing wide-spread impact, but we apologize nonetheless to those who are affected. We’ll update this status post again when we have news about this situation.

Update October 30th 12:08pm PDT: We have found and eliminated most of the cause for the service problems, but some still lingers. Again, apologies for the inconvenience this may be causing, and we’ll update this post when we have more information.

This entry was posted on Friday, October 25th, 2013 at 1:22 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.7 Major Release Now Available
4:28 pm

WordPress 3.7 Major Release Now Available

Posted (October 24th, 2013 at 4:28 pm PST) by Mika E.

It’s only been a couple months since 3.6, I know, but here we go again!

The newest version of WordPress (3.7 aka “Basie”), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

3.7 is a sneaky release, most of the cool things are behind the scenes so you may not actually notice a huge change. That’s okay!

For more details, you can check out the release notes here, with further details available in the WordPress codex. Heck you should check the writeup on poststat.us too!

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.7 in the DreamHost forums, and the 3.7 master list at the WordPress.org forums, where issues relating to upgrades are being tracked and updated as they come up.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

Oh and in the manner of Steve Jobs, one more thing. Minor releases (like 3.7 to 3.7.1) will now be automatically applied by WordPress. We won’t be disabling this as we have absolute faith in this feature. That said, we’ll still run our One-Click updates for everyone, just in case the automatic tool doesn’t work (or is disabled). But you’d never disable that, right?

This entry was posted on Thursday, October 24th, 2013 at 4:28 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “WordPress 3.7 Major Release Now Available”

网站挺不错,有空来我的网站看看

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West Data Center (Los Angeles) Core Router Configuration Update – Wednesday October 23rd 10pm PDT (RESOLVED)
10:49 am

Network Maintenance US-West Data Center (Los Angeles) Core Router Configuration Update – Wednesday October 23rd 10pm PDT (RESOLVED)

Posted (October 22nd, 2013 at 10:49 am PST) by Ralph

On Wednesday October 23rd starting at 10pm PDT, we will be conducting maintenance on our core routers in our US-West Data Center (Los Angeles). The window for this maintenance will last for 4 hours, till 2am PDT, Oct. 24th. During this maintenance, we will be reconfiguring our two core routers to optimize support for IPv6. Our maintenance plan will allow us to reconfigure our routers without causing significant impact on the services we provide. We will also allow for time to monitor the system to ensure that there are no unexpected issues.

The overall expected impact of this maintenance should be unnoticeable, however, we will open up comments and monitor support throughout the window.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services. Comments will be turned on during the maintenance to assist you in real time, should we experience any unexpected difficulties.

Update October 24th 12:00am PDT: This maintenance is now complete. Thank you for your patience. We are happy to announce this maintenance was completed with no noticeable impact.

Update October 23rd 11:55pm PDT: Step 4 is complete. We are monitoring to make sure everything is okay so that we can close the window.

Update October 23rd 11:35pm PDT: Step 4 is underway now. This step is similar to step 2 earlier. We do not expect any issues whatsoever. We will continue to monitor. When this completes we will be done with the maintenance.

Update October 23rd 11:30pm PDT: Nearing the end of step 3. We are at a point where there will be absolutely no impact, on this step. One more step after this and we will be wrapping up this maintenance. We are on the stretch run.

Update October 23rd 11:15pm PDT: Moving along on step 3 and we are not seeing any issues. If you notice any impact, please wait 10 seconds and try reloading your site. Any service interruption has been very brief.

Update October 23rd 11:00pm PDT: We are now well into step 3 of the maintenance. We have yet to see any noticeable impact to services.

Update October 23rd 10:50pm PDT: Step two has completed with no impact. We are moving on to the third step.

Update October 23rd 10:40pm PDT: Everything has checked out okay with the first step. We are now moving on the step 2 of the process. This has an even smaller expected impact from the previous step. We will continue to monitor as we progress through this step.

Update October 23rd 10:20pm PDT: First step of is complete. This is a 4 step process. We are now making sure everything is working as expected before beginning the second step of the maintenance.

Update October 23rd 10:15pm PDT: So far our initial step is going smoothly without any incidents. We will continue to provide updates as we successfully complete each step of the maintenance.

Update October 23rd 10:05pm PDT: First step of our maintenance is underway. The expected impact is extremely slight and should go largely unnoticed. If you feel you are affected please comment!

Update October 23rd 10:00pm PDT: Our team is assembled and ready to start this maintenance. Please note that we do not expect any noticeable impact throughout this maintenance. We will however stand by to assist you if you feel you are affected in any way.

This entry was posted on Tuesday, October 22nd, 2013 at 10:49 am and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading from PHP 5.2
12:43 pm

Upgrading from PHP 5.2

Posted (October 16th, 2013 at 12:43 pm PST) by mir

PHP 5.2 has been defunct for some time (it was officially moved to ‘unsupported’ January 6, 2011) and thus is very outdated. In an effort to help secure your sites and data we’ve made it easier than ever to upgrade your PHP version by adding a feature to the panel here:

https://panel.dreamhost.com/index.cgi?tree=home.over&

which will identify domains which have PHP 5.2 running and let you upgrade them with just a couple of clicks. Please take the time to check this out and make sure you are up-to-date. You may also want to read this article if you have questions about upgrading:

http://php.net/manual/en/migration53.php

Please note that it is still possible to revert to PHP 5.2 at this time if it causes any issues with your site (you may just need to upgrade your applications which is also very important for sound security practices).

This entry was posted on Wednesday, October 16th, 2013 at 12:43 pm and is filed under DreamHost Features, Policy Changes, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

26 Responses to “Upgrading from PHP 5.2”

I tried to go to the link and it is just my usual control panel … I’m missing whatever you’re pointing to here … or does it mean I’m already current?
I think I’m current but it would be cool to check … because clearly I’m an idiot :)

https://panel.dreamhost.com/index.cgi?tree=home.over&

We host quite a few “legacy” websites on Dreamhost that DO NOT WORK with PHP 5.3

Clients do not want to pay to upgrade the code.
I hope we will be able to continue to use 5.2 ??!!

@Ann Martinson:
control panel
-> domains
-> manage domains
-> (next to your domain name – see “Web Hosting” column -> click “edit”)
-> look under “Web options” section
-> PHP mode.
All the best. :)

“PHP 5.2 has been defunct for some time”

Ok great and what about Ruby 1.8.7? Why is it still alive and shinning with all glory on the shared servers???? When will you support Ruby 1.9.3 or the new 2.0 or at least RVM support? And please don’t tell me install RVM, it requires sudo powers to install some packages and it’s necessary to use fcgi.

I tried to upgrade a few of my 5.2 sites to 5.3. They becomes extremely slow. Is this known issue?

I cannot get into my web panel nor can I access my website. My uploader won’t let me upload!
What is going on? Do I have to find a new Webserver? I need my websites to work!!!!!

can not view any of my domains

@Sharon Wick and generally:

I have similar experience as Sharon. Would you suggest, e.g. provisionally, to move back to PHP 5.2?
I checked, I use PHP 5.3x FAST CGI.
Also, have installed plugin W3 total cache, using wordpress enrvironment. This plugin used to give a bit better results (it seeemed to me) compared to WP Super Cache plugin.

Maybe, would you suggest different cacher? Any info, suggestions?

Nevermind php, email is massively screwing up again.

I’ve had poor reliability from all my pop3 email accounts for some time now. Last time I contacted support about this, the response I got was to use IMAP instead.

Well I’m sorry, but Dreamhost advertise that they provide both IMAP and POP3 email, so I expect them to BOTH work well.

Totally screwed at the moment by this upgrade. It kept prompting me to do it whenever I logged into my control panel so I upgraded all my domains as suggested. But now I am finding that some PHP plugins which are supposed to be installed by default (ZendOptimizer, ZendLoader and ZendGuard) are not installed properly on the new version and the Dreamhost wiki instructions for this are hopelessly out of date and do not reflect the current Dreamhost configurations. Rolling back doesn’t seem to help and have been waiting 9 hours for an urgent request to support with no reply so far. Meanwhile my company is down, unable to accept any payments…

All of my sites crashed right before the upgrade. I came to dreamhost to find out why, saw the upgrade and thought maybe that was it. Upgraded.
All sites still down.

I think something else happened this morning and that some of the problems we’re seeing aren’t related to the php upgrade but something else. I can’t even get a simple HTML file to load (something that doesn’t call/use PHP).

“Upgraded” PHP and WP this morning. When I try to get to my WP login I see this at the top of the screen

Warning: Cannot modify header information – headers already sent by (output started at /home/pnuesslein/insidetheperimeter.net/wp-content/plugins/gravatar.php:2) in /home/pnuesslein/insidetheperimeter.net/wp-login.php on line 409

Warning: Cannot modify header information – headers already sent by (output started at /home/pnuesslein/insidetheperimeter.net/wp-content/plugins/gravatar.php:2) in /home/pnuesslein/insidetheperimeter.net/wp-login.php on line 421

and am being told that I need to enable cookies (already enabled) in order to log in.

Little help, please?!?!

@Ann Martinson – All your domains are already using PHP5.3 so no upgrade is needed, you’re good

@Dave – The same argument was used for PHP4 and Windows3.1. Unfortunately sometimes that software just needs to get upgraded. For the time being PHP5.2 is still offered on our servers. It’s highly unlikely it will be supported forever though. This might help!
http://php.net/manual/en/migration53.php

@Carlos – Our shared servers are all currently running Debian Squeeze, the Ruby version for that release version was 1.8.7. In the next few months we’ll be upgrading all servers to Ubuntu 12.04 Precise which should give us more options.

@fishy – PHP5.3 is generally considered to be quicker than PHP5.2. It’s hard to say why you would be seeing significant slower response after that change.

@Sharon Wick and Mark Green – Please contact support through the contact page on DreamHost.com http://www.dreamhost.com/contact/ They can assist you with accessing your accounts.

@yanoya – It’s surprising that you would find PHP5.3 slower than PHP5.2. Both WP Super Cache and W3 total cache ought to work well, though I haven’t done any benchmarks for each using both PHP5.3 and 5.2.

@Doug – If necessary you should be able to downgrade to PHP5.2 through the Domains > Manage Domains section of the panel. What you mentioned is one of the biggest issues we’ve seen with upgrading to PHP5.3. http://stackoverflow.com/a/11584674

“Zend Optimizer doesn’t exist for PHP 5.3. The product was renamed ZendGuard when it was released for PHP 5.3. However, ZendGuard cannot load files that were encoded for Zend Optimizer, and vice versa — they’re effectively two separate products. If you’ve got an application that was encoded for Zend Optimizer, you’re stuck on PHP 5.2.”

That is the reason we didn’t install those packages by default. If you wish to add ZendGuard to PHP5.3 there are instructions here. http://wiki.dreamhost.com/Installing_php_loaders

@Raven – Your issue sounds unrelated to PHP versions. The version of PHP installed on your site shouldn’t impact static html files.

@sdcyspgyio – I agree, but have you considered jirfijfrjewweo?

@PJ – It looks like you’ve disabled that plugin now, though looking at the files I don’t see anything that ought to cause problems. You might check if there is a newer version of that plugin available.

since moving to php5.3 sendmail() seems to have stopped working, anyone else having this problem?

Unfortunately, there’s still no support for nginx and PHP 5.3, after a LONG time of waiting and asking about it. :(

The upgrade is causing all kinds of problems with all of my websites – slow response times, freezes, ‘no data received’ and ‘connection reset’ error messages. It’s been a nightmare.

Just when things were running smooth as silk, Dreamhost had to throw a monkey wrench in it and screw it all up!

NONE of my webistes, four of them, have been accessible for nearly a week. I’m ready to take legal action. I can’t even access them to move to a different hosting business.

NO sites accessible for about a week (not even so I can move them to a different hosting company)! Going to take legal action. Want compensation from you. Dreamhost is affecting our livelihood and ability to pay bills.
padillatree.com
msgreenclean.com
yonkerspottery.com
corimorenberg.com

Dreamhost is not updating their status with news about the most critical downtime that many of us have ever experienced. Please post what server you are on if your issue is not PHP-specific. They claim that YEREVAN had a RAID meltdown and is being restored. I’m not sure how that can take 5 days – this happened in the AM of Wednesday November 6th. My site is my primary source of income and today is the fifth day where I haven’t made a dime.

I have a custom install of PHP running on my account, so PHP migration at Dreamhost has never affected me in the past and shouldn’t be now.

@Dan_J Nice job responding to everyone!

My site has now been down for longer than 6 days. Lots of email exchanges with support with only form-letters back. This major server outage isn’t even reported on dreamhoststatus – why not? Easy fix: move my web site to a different server by editing your DNS records and I will take care of uploading the backup from my end.

I’m on YEREVAN as well and it’s been down since Wednesday. Please give us least give us some clue of time frame, even it’s half done would suffice. At least then I can determine if I need to move my Dev sites to another server too continue working. It’s taking “longer then expected” doesn’t help.

Move your sites now to either a different server or a NEW USER on Yerevan. I have multiple sites hosted and only my primary and commercial site was still having issues. Today I spent all morning and afternoon creating a new user account to host the site. Do NOT use DM’s automatic copying feature or you will get screwed up permissions and it will take more than a day to set up. Make sure you have up to date local copy of everything on the site, then use DM’s domain control panel to run the site as a different user on a brand new EMPTY path.

FTP to that new user once it’s set up (supposed to be 10 minutes but more like 30-40 minutes right now). Then upload your local files to the new user account. It might take another hour or more before DM’s DNS reflects the new path, but eventually it’ll work.

This has truly been the low point in Dreamhost’s service in over 7 years. They don’t even have this issue listed here on the status site and it’s been a problem since last Wednesday with no signs of being resolved. Dreamhost staffers should have offered immediately to modify DNS records to set users up on different servers if they wanted to do their own site restore via FTP, especially since it was obvious that their own “restore” process was going to take longer than half a day. It’s been almost 7 days now!

Over the next days and weeks I’m going to set up an account with another ISP to act as a mirror – maybe act as the primary with Dreamhost as the mirror. This way it will be a lot quicker to change over (fail-over) should something this catastrophic happen.

Since 5.3 all my sites run significantly slower, including phpmyadmin

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Alert: NextGenGallery v2.0.23
3:33 pm

Alert: NextGenGallery v2.0.23

Posted (October 14th, 2013 at 3:33 pm PST) by Mika E.

Last month, NextGEN Gallery pushed a release that had a flaw in how it handled temporary data, causing users to suffer database issues. If you’re still using version 2.0.23 of the plugin, we urge you to upgrade at the earliest as this will correct the issue and clean out the database issues as needed. NextGEN Gallery worked closely with core WordPress developers to resolve this issue, and are aware the 2.0 upgrade hasn’t been at all as smooth as they would have liked.

For more information on the recent changes, see the NextGEN Gallery Changelog or visit their blog to read about the recent changes.

You can upgrade NextGEN Gallery via your WordPress Dashboard.

This entry was posted on Monday, October 14th, 2013 at 3:33 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Alert: NextGenGallery v2.0.23”

Hello Admin,

My vps keep returning “Service Temporarily Unavailable” error message when accessing database.
Is there any problem with dreamhost?

Regards,
Bowo

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance – Router Redundancy Improvements and Data Center Maintenance – Monday Oct. 14th 10pm US-West (Los Angeles) (RESOLVED)
2:43 pm

Network Maintenance – Router Redundancy Improvements and Data Center Maintenance – Monday Oct. 14th 10pm US-West (Los Angeles) (RESOLVED)

Posted (October 11th, 2013 at 2:43 pm PST) by Ralph

This Monday Oct. 14th starting at 10pm, we will be conducting maintenance to reestablish redundancy, which was impacted by a hardware failure last week. During this maintenance we are expecting to see a few 30 second interruptions of service. These network interruptions should be unnoticeable.

Once the router maintenance is complete, at around 12 midnight, our data center will then be conducting another maintenance. This maintenance is similar to Thursday’s maintenance, which was completed successfully in roughly 30 minutes. We will not be conducting this part of the maintenance, but we will have our team on hand to make sure everything goes smoothly, as this data center maintenance does affect our network. The data center maintenance is expected to have no impact on the services we offer.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services. Comments will be turned on during the maintenance to assist you in real time, should we experience any unexpected difficulties.

Update Oct. 15th 12:45am PDT: This maintenance is now resolved. Our team monitored the changes to make sure services remained working. We have not seen any issues. We will have our monitoring staff keeping an eye on things, but this maintenance is now considered resolved and completed with 0 customer impact. Thank you for your patience!

Update Oct. 15th 12:36am PDT: Second link is now up. We are now testing and monitoring to make sure everything is working as expected. We have not seen any impact due to this maintenance.

Update Oct. 15th 12:35am PDT: Second link is now down and being moved.

Update Oct. 15th 12:32am PDT: First link is transferred correctly. We are monitoring it as the second one is prepped for move. So far we are not seeing any issues.

Update Oct. 15th 12:30am PDT: First of two links is now being moved in our data center. We have a redundant line up to make sure that no interruption occurs during the maintenance. As we get updates we will let you know.

Update Oct. 15th 12:02am PDT: All data center technicians are on hand to start their part of the maintenance. We are not expecting any issues since we will be using a redundant connection.

Update Oct. 14th 10:45pm PDT: The router redundancy part of this maintenance is now considered resolved. We will next update once it is time for the data center maintenance to start.

Update Oct. 14th 10:30pm PDT: Router redundancy is now complete. Second part of this maintenance will be run by our data center. They are scheduled to start around midnight. We are still monitoring the re-installed router to make sure that there are not any issues.

Update Oct. 14th 10:20pm PDT: The initial step in the two part process of getting our redundant core live again has completed with no noticeable impact. We will move on to the next step which will involve moving some networks to live on this core to ensure it is working correctly. If it not working correctly there could be some service disruption, however we will be able to recover quickly if we notice any problems.

Update Oct. 14th 10:00pm PDT: Our team is ready to execute the first part of tonights maintenance. This will be the task of re-establishing redundancy to our router. There may be a slight network disruption during this step of the maintenance.

This entry was posted on Friday, October 11th, 2013 at 2:43 pm and is filed under Data Center Maintenance, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Network Maintenance – Router Redundancy Improvements and Data Center Maintenance – Monday Oct. 14th 10pm US-West (Los Angeles) (RESOLVED)”

Sever ds2032 has been down for over 2 hours now…

@shawn grossman – there are no services affected by this maintenance. I have however ping’d the support team to check on your dedicated.

@shawn grossman – our operations team is checking on your machine, it might need a physical reboot. Support will get back to you asap.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network maintenance – upstream provider and data center – US-West Data Center (Los Angeles) Thursday Oct 10th 10pm PDT (RESOLVED)
11:09 pm

Network maintenance – upstream provider and data center – US-West Data Center (Los Angeles) Thursday Oct 10th 10pm PDT (RESOLVED)

Posted (October 9th, 2013 at 11:09 pm PST) by Ralph

On Thursday October 10th starting at 10pm PDT, there will be a maintenance at our US-West (Los Angeles) data center. This maintenance will be conducted by one of our upstream providers and the data center staff.

To ensure this goes smoothly we will also have our own team on site to help assist the providers in any way possible, or to respond to any emergency situations that may come up.

This maintenance involves moving network links from one physical location to another. The links that are being moved are redundant, and will be moved one by one. Even though he services from this upstream provider will be affected by this maintenance, we don’t expect this to have any impact on customer services. Regardless, we will stand by, so that we can communicate if anything comes up. Comments will be opened once the maintenance begins, so that we can assist you with any possible related issues and concerns.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services.

Update Oct. 10th 11:00pm PDT: This maintenance is now resolved. There was 0 impact to the services we offer. If you are noticing any issues, please contact our support team.

Update Oct. 10th 11:00pm PDT: The data center techs are currently getting ready to start the maintenance. We expect 0 impact from this maintenance. As we get updates we will update this post.

Update Oct 10th 10:00pm PDT: The start time for this maintenance is expected to be in roughly one hour. Please keep in mind that this is not a maintenance we are performing, but one run by our data center and one of our upstream providers. We will post updates as we receive them. We do have staff on hand in the data center should anything unexpected occur.

This entry was posted on Wednesday, October 9th, 2013 at 11:09 pm and is filed under Data Center Maintenance, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webmail Connection Errors ( RESOLVED )
7:03 am

Webmail Connection Errors ( RESOLVED )

Posted (October 9th, 2013 at 7:03 am PST) by Dan J

We’re currently investigating errors being reported by the webmail.dreamhost.com servers. This is impacting webmail connecting to all email servers. Direct IMAP/POP/SMTP connections from an email client such as Thunderbird or Outlook also appear to be experiencing intermittent connections issues. We should have more info shortly.

Update: October 9, 2013, 11:03 am PDT: We are happy to report that the mail service has been stabilized. You should no longer encounter any issues when accessing your email. We sincerely apologize for any inconvenience this has caused. If you are still seeing any kind of mail related issues then please contact support.

Update: October 9, 2013, 10:14 am PDT: We are continuing to see vast improvements. Our system engineers are continuing to actively monitor the stability of our mail service. Thanks for your patience and understanding. We will continue to keep you updated as the improvements progress.

Update: October 9, 2013, 9:21 am PDT: There has been improvement. However, a few mail machines require the attention of our admin team. There may be intermittent connectivity to the mailserver until the machines are stabilized. Our admins are diligently working towards having the machine operate at optimal performance. We sincerely apologize for the inconvenience this has caused. Your continued patience is greatly appreciated. We will continue to provide updates as information is made available.

Update: October 9, 2013, 8:36 am PDT: All our tests appear to confirm that webmail and mail in general is now functioning properly. We apologize for the problems this morning. If you are still seeing any kind of mail related issues then please contact support.

Update: October 9, 2013, 8:30 am PDT: It appears that the webmail and the IMAP/POP/SMTP issues are no longer preventing access to mail. We’re currently running tests to confirm.

Update: October 9, 2013, 7:50 am PDT: It looks like the IMAP/POP/SMTP issues are related. Our systems engineering team are still investigating. Thanks again for your continued patience!

Update: October 9, 2013, 7:25 am PDT: We’ve seen some complaints of direct IMAP/POP/SMTP connection issues as well. That may be related. At the moment we are continuing to investigate and rule out causes. Thanks for your patience!

This entry was posted on Wednesday, October 9th, 2013 at 7:03 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

226 Responses to “Webmail Connection Errors ( RESOLVED )”

Yeah, I’m having issues using mac mail / imap. port 993 not connecting.

Unable to connect via IMAP/SMTP. Submitting a ticket.

I’m not able to connect using Apple Mail. Hope this is resolved shortly.

It is impacting IMAP. Can’t send using ‘mail.mydomain.com’, and changing to another outgoing SMTP works, but can’t save the sent email into my IMAP Inbox.

My Outlook hasn’t worked all morning either. Seems to be a broader problem.

Connections from Outlook are not working , tested on several accounts.

IMAP totally broken for me too with the usual port-993 errors. However, this has been going on for weeks (with the promise that it’ll get fixed), so maybe it’s unrelated.

We’re not able to receive or send using direct IMAP/POP/SMTP across various platforms – Outlook, MacMail, etc. Submitting a support ticket.

I cannot access mail for bestoptiontrading.com from my iPhone. I also cannot log into the webmail.

October 9th, 2013 at 7:07 amVaughan Schmidt Says:

Also affecting email clients at least on sending SMTP ( using homemail-sub4)

I am not getting email on either desktop or phone using Mac Mail.

Yes, there are problems with POP as well. I mentioned this in my ticket sent at 6:31 PDT before I issued a second ticket telling you that webmail was not working either at 6:57 PDT.

Hurray! Thank you for updating and using the DreamHost STATUS page. I really appreciate knowing that you have acknowledged that there is an issue and that you are letting your users know.

All the best!

Connection resets when trying to login to Roundcube, although the login page loads fine.

Outlook. not able to get mail…

Connections to all POP Email accounts on homiemail-sub4 are slow and/or failing. I am able to check mail on other servers, just none of the accounts hosted with DreamHost.

We are continuing to have unacceptable email outages. At the moment, we are unable to connect to our email service via IMAP. What is the status of the email system (homiemail host?)? We have had numerous delays and outages over the past couple weeks on this server and our users are very frustrated. We cannot continue to run a business with email outages!

Also having trouble connecting via SMTP (port 993) from talitha-borealis.dreamhost.com (my Joomla! back-end is unable to connect to send system emails). Submitting a ticket as well.

All mail is down

POP NOT WORKING EITHER.

TIME TO FEED THE HAMSTERS, DREAMHOST.

Mail issues – says cannot connect to mail server … ! ! !

My users cannot connect through POP/IMAP

we’re on homiemail-master

Still problems with both apple mail and roundcube.

email isn’t coming in through outlook or any other form of email client….

October 9th, 2013 at 7:10 amAndy Ringsmuth Says:

It’s MOST DEFINITELY affecting POP/IMAP/SMTP all around.

October 9th, 2013 at 7:11 amAndy Ringsmuth Says:

I should add – all incoming mail is being deferred or spooled. I can tell that from my spam filtering provider.

Definitely IMAP/SMTP issues also. Independently verified by Pingdom, not a local/ISP issue.

all email is down. I already submitted a ticket.

We’ve been having issues over POP/IMAP and SMTP on at least two accounts that I maintain for at least 1 hour. I’m sure others are in trouble just haven’t reported. Already sent ticket 30+ min ago, but this is becoming way too frequent and no long term solutions are being provided.

Yes definitely affecting IMAP all round. None of our emails are working through either IMAP in Outlook nor webmail.

As a matter of fact, there IS a problem connecting through IMAP. It was why I reported this in the first place about an hour ago…

also having trouble on outlook AND thunderbird – tried webmail as last resort and that is not working either.

This has become a joke now. We have logged tickets every 2 weeks with our email services have one problem or another. The support team and tech team are too busy creating new products to give a damn about the products and customers that already paid for their services.

problem to send and receive e-mail via thunderbird

Yes, POP is down, so is Webmail, and our customers have begun ringing us! Any ETA on this?

i’m on homiemail-master, nothing works (imap/smtp/pop/webmail)

mail.app and webmail both down

Mail in general is affected, POP3 and IMAP… multiple domains.

ps12997 VPS

C’mon Dreamhost, get our mail servers back up! Sigh.

Problem exists with Thunderbird 2.0.0.24 from Windows 7.
“Connection timed out, to sub4.mail.dreamhost.com”

Excuse me, some improvement while I’m typing, its now logged but very slow and Download message 1 of 145″ seems stuck. 145 seems a bit high to have arrived in the past hour. Local time 22:12 with last email received 21:03.

Webmail getting a 503 error and POP through Mac Mail is down on several DH accounts, on different servers.

IMAP and POP , WEBMAIL ALL DOWN

All our company email is down as well using Apple Mail. Both IMAP/POP accounts.

I love that when I reported the ticket, it acknowledged the problem and asked me how I wanted to be kept up to date…via email. Ah..the irony. Just made me smirk a bit.

Mail not working on Mac Mail, iPad or iPhone – no IMAP access

I can confirm that it IS affecting IMAP. Send/Receive errors all around. Please fix ASAP!

Hi we need an ETA on this Dreamhost – as our clients are ringing and quite rightly want to know when it will be ok.
Please let me know asap!

Using outlook and emails are not working…

Unable to connect via IMAP/SMTP. Submitting a ticket

Webmail is not working. please help

Email is not sending nor receiving via Outlook. Please advise on timeline to fix. Thanks.

Once again, all my mail and that of my clients are DOWN.

NightmareHost at its best is becoming very unstable to continue to shell out money each month. I’d get better service from lesser companies.

C’mon DH, let’s get this resolved NOW

If DH isn’t reading the comments here, what’s the point of leaving them?

homiemail
Yeah, I’m having issues using mac mail / imap. port 993 not connecting.

Does anyone else’s mail go offline on a daily basis? I have to go around and “take accounts online” on all of our computers at least once a day. We use Apple Mail both POP and IMAP if that makes a difference. I’m getting very sick of fixing email for all of our business computers every day.

are you freaking kidding me! AGAIN!

All emails are down across multiple domains, right at the moment when reliability is crucial for our org.

i am not able to connect via webmail either

does not answer the ip 208.97.132.231

IMAP & POP not function on multiple domains across all clients.
No function http: or https:/webmail.xxxxx.xxx, or outlook client etc.

Can you guys do your job and fix this. I have problems every day with your service. This is a joke and unacceptable!

No access through any version of Outlook. Squirrel Mail not working. We are on Homiemail Sub 3. OMG the government shutdown has hit Dreamhost!

We can’t use our emails right now. This is really frustrating.

can’t connect on phone or web.

IMAP, POP and WEBMAIL all is down!

Emails not working… please solve this ASAP

I HAVE SERIUS PROBLEM IN MY BUSINESS MAIL, HERE IN BRAZIL IS MIDLE OF DAY WORK, PLEASE GIVE A IDEA HOW MUCH THIS GO AWAY. THANK YOU.

Cannot retrieve email either via iPhone, iPad or mail app in OS X. Squirrelmail is also out.

Really annoying that the first thing I checked was dreamhost status page, before submitting my ticket. In my ticket I indicated imap wasn’t working either. 20 minutes later, there is finally and update on dreamhost status, but it says “we believe imap is working” – then I read comments that others reported this issue hours ago. You guys have got to address system wide issues promptly and disclose when there is a problem.

all of our mail is down. webmail and direct imap/pop3
the issues started late last night and went down completely this morning

get this fixed ASAP we are at a halt without our email system.

My client reports general e-mail failure when trying to log in. IMAP errors reported, not connection to Outlook.

No luck with either webmail or Mac Mail (IMAP). Webmail show the following:
Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Outlook not receiving mail – this has been going on all week. When we do receive it is hours late. Just sent 3rd request for a fix.

Hello!
all mailbox are not working. (webmail, smtp/imap etc)

Any news?

October 9th, 2013 at 7:23 amMatt (in Indiana) Says:

Same here… no webmail… no t-bird. This is becoming a MAJOR problem with Dreamhost– their email issues are way too frequent.

No mail via webmail neither IMAP or POP. I tried with Mail in Mac.

Cannot connect using Apple Mail. Hope this is resolved shortly.

Here POP/SMTP is fine and only webmail not working…

It’s definitely causing problems in Outlook——can’t get any mail.

Not able to connect via Mail in Mac, and webmail via browser. Need fixed ASAP. Thanks.

No POP3 access to my mail accounts from here (Germany, 16:30 Local Time)

:-(

Dreamhost sucks…looking for another service…

what time to normalize? (Brazil)

All email is still out for us using IMAP and webmail. Obviously we need this back ASAP.

Not working as well…

Down again!!! We have a bussiness to run and this has been getting excessive this year. This is close of payroll and our people cant get there time cards sent in

Apple Mail is not working! Webmail also not working!

Emails that should be forwarded are not getting forwarded either. So it looks like receiving email is down too, not just connecting to mailboxes.

Come on DH!!!
This is becoming a habit….

POP3 and SMTP is not working!! please solve ASAP

TPS.

My Outlook Pop mail is also not working, urgent emails to get out this morning……

Outlook hasn’t been working all morning. No incoming, outgoing emails.

Problem beging yesterday on outgoing emails, I´m connecting from Chile.

Forwarding-Only addresses are not functioning.

Hola estimados amigos no funciona el servidor de correo ni con outlook ni con thunderbird por favor ayúdennos solucionando el problema. Gracias

Still cannot connect.

no pop3 no imap no webmail

This is not the first time!
It happens every week at least once!
I’m going to transfer all my sites to another company!

Forwards aren’t working either. I’ve sent a couple test messages and they haven’t made it on their way.

cazzo!!!!!!!!!!!

Go DreamHost! Don’t make your Happy Dreamhoster become a sad DreamHoster!

Out of email since 14:00UTC

Thank god I’m in the process of moving to Google Apps for email.

Our customers are reporting that they cannot send/receive on phones/outlook/webmail.

webmail, pop and imap down…eta?

I too am having issues using mac mail. Over the past several months I found an increasing large amount of problems with your service. What is going on? I am n the verge of changing all account to another service. Lets get this fixed ASAP PLEASE!

Here’s the error message…
Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

I cant send email, somebody shot me squirrel and me pigeons are too lazy.. won’t somebody think of the incommunicable children!

IMAP/SMTP connections timing out on port 143. Have been having temporary issues in office for several weeks but this is the first time it’s been down for everybody for an extended amount of time.

Thanks for your work/response on this DH… will keep checking back.

Here POP/SMTP is fine (sub5.mail.dreamhost.com [208.113.200.129]) and only webmail not working…

DH obviously doesn’t get it that email is mission critical for everyone. the only solution is find another host as this is regular problem. we are leaving ASAP.

Also having issues with POP3 sending/receiving email via Outlook, US Connection. Please post when this issue is resolved

My dreamhost email accounts are not working for my business. None of them…since 9:35am Eastern Standard Time. I appreciate that DREAMHOST has acknowledged this problem. At least I know it’s not me or my computer. I will be patient for a fix although granted I’m a little nervous being that my entire business runs here and missing emails is somewhat critical for me. I will survive if fixed today.

Thanks!

HELP!!!!!!

Problems with Thunderbird IMAP and Outlook Pop3.

Help. Employees with cut neck. dismissed.

Mail for both Windows, droid and ios clients not working (POP and IMAP). Client would like to know when this will be fixed.

This is unacceptable. Dreamhost e-mail issues have been occurring too frequently and is costing businesses to lose productivity and ultimately money.

Don’t think this is an ongoing issue? Read the history. Loads of e-mail outages. It’s incompetent and unacceptable.

Problems retrieving emails. Thanks.

no email for pop or smtp.

What is up with all the DH mail problems these days?

So…is email down because of Obamacare?

Having problems here too

October 9th, 2013 at 7:43 amJeffrey Hatfield Says:

All four of our offices are down for over three hours! Can we get an update please?

If it helps, my webmail and outlook on PC aren’t working, but I can still connect via my android phone… odd…

My Mail is down as well, using Mac Mail IMAP and on the iPhone. Really embarrasing when this happens and clients are waiting on my response to earlier emails. Looking forward to having it back up and running real soon.

Cry me a freaking river people. Let the smart people fix the hardware.

October 9th, 2013 at 7:43 amWillians Reis Says:

All webmails subdomains are down

All email down (IMAP, webmail). This could not have come at a more crucial time for me (but of course they are all crucial).

Webmail error:
503 Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

IMAP client:
Just hangs.

1 hour + and still no work around or back up system in place for us DH customers.

P.S. I like how DH puts up a forum for us to vent but clearly states that
Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost

Submit an email support ticket – OMG that is rich

Also getting error using mac mail / imap. port 993 not connecting.

My POP email accounts are not working through Outlook or through my email program on my cell phone.

Also, the webmail is not working through DreamHost

Is this related to the gov’t shutdown? This country is falling to pieces!

Exchange is looking better and better cost saving using dreamhost dont do anything for us if there are continous outages like this. This past year has been horrible

Fight and you may die. Run and you will live at least a while. And dying in your bed many years from now, would you be willing to trade all the days from this day to that for one chance, just one chance, to come back here as young men and tell our enemies that they may take our lives but they will never take our freedom!

October 9th, 2013 at 7:47 amfrustrated customer Says:

seriously, FUCK YOU FUCK YOU FUCK YOU FUCK YOU dreamhost! no … sorry nightmare host!

Unable to log on to squirrel mail, no mail has been sent or received since around 21:30 CST 10/08

so now my phone is showing that I have 2,147,483,702 unread emails!!!!!!

Ditto… Need email please.

This is nothing compared to the outage that lasted a couple days and haunted for weeks a couple years ago in January I believe.

POP email not working. Webmail not connecting at all (503 error). Good thing I don’t have a deadline today, like I did last week when my email went wonky!

Email is down. Just like everybody else’s, apparently. Lots of email problems of late… just saying…

pop3 def affected, none of my emails work – im with homemail-sub3
no squirrelmail either.

This is a big problem. Outlook is not working either. Everything is down.

YES, there is s F!@#$%^%ing problem again with your F!@#$ING email system…another week another problem…..TIME TO MOVE

Outlook not pulling up email, just like everyone else right now. Losing business by the second if we cant get it back.

Same issues here connecting through Outlook and iphone. No mail connection happening.

My IMAP is fine, my wife’s is down. Both Dreamhost, different domains. I blame the NSA.

@SH you might have issues with your keyboard

It’s definitely not just webmail. Nothing coming through on POP and IMAP, using Mail and Outlook on Mac.
Must say this is happening a bit too often recently.

No email here for the entire business, I am not happy.

@Hal your wife’s what?

e mail is not working!

Dreamhost stoped caring a long time ago. Every Monday, Wednesday, and every other day the mail is down for hours. This is a recurring issue. My bro and I have some old domains on DH but I’m definitely going to move them to another Hosting.

Any ideas on resolving the issues? I am losing business every minute this continues.

@Brian “email”

No email for the past two hours.

@Brian “email” or “e-mail”

Help I need my email now… I am in the middle of a large RFQ please!

Having problems with our email via Outlook as well. Telling me it will take 24 minutes to download 200KB of mail and then, ultimately, I get an error message.

Bring me the cheese!

tired of this

LEAVE DREAMHOST MAIL …. a pigeon is more reliable

I kinda need my email people. Please hurry and fix this…

Saaaame Mac Mail IMAP 993 Timeout GAH!

Email will not send or receive.

Not able to get email via Thunderbird. What year is this 1996?

In soviet Russia, email sends you

30 min to find out they are related?

It’s 10:00 am CST and company-wide, all our mail is down. Using Macmail pop accounts on work computers and IMAP on laptops and iphones. Boo.

Our emails are not sending or receiving. Please let us know status of this problem.

Dreamhost… hopefully you call can get this fixed soon… I won’t complain like the others… your services have done me well for many many years. It’s understandable… there will always be unexpected problems when it comes to computers and IT infrastructure.

However, have to chuckle at all the people commenting that say “losing business by the second” and “our entire business runs on this” and “our business is going to go out of business and we will all die if you don’t get this fixed this second ZOMG!!!” I mean, really? If you’re running your entire business on a webhost that you probably pay less than $10 a month for… you’ve got bigger issues. If you really know how to run a business, you should have backups and contingency plans in place… or you should probably subscribe to a much better service at Dreamhost and not <$10/month shared hosting. I'm pretty sure Dreamhost offers dedicated servers, which is much more conducive to "running a whole business on." Oh, and "my email has been down for an hour, I'm switching hosts!!!"… go for it… you'll have WAAAAAY more downtime, and much more pain and headache just TRYING to move to a different host… give me a break! Give them a break… crap happens… they're smart people… they'll get it fixed as soon as they possibly can… not like they're taking the morning off to go catch Gravity on IMAX 3D.

I thought our email seemed quiet today. No luck with IMAP / POP or webmail.domain.com, either. :-(

Sending good vibes your way – hope this gets resolved quickly!

October 9th, 2013 at 8:04 amDavid Kittredge Says:

No email. Again. On any device, any login.

Dear, we have problems in our emails. We can not send or receive emails and this is generating difficulties in selling our products. We please check urgent problem. Thank you.

We’re starting to get mail, but some mail is missing.

Mail server is down! please fix it !!

Email is down…NOT HAPPY! Been having lots of issues lately with email connections both sending and receiving. have not been happy with the constant tech problems.

I’m switching to Exchange! Because I like wasting money! EXCLAMATION MARK AHOY!!!!!!!!!!

Webmail still offline. Have seen a series of outages over the last few weeks. The symptom has been to log in to Webmail and be bumped after 1 or 2 operations with a request to log back in. Truly annoying.

Hope you all get it fixed before happy hour! Or else how are we supposed to notify people where to go for happy hour this afternoon in New York City??? (ps, get it fixed, and then you all can go to happy hour! :)

I was able to Send and Receive two email accounts via POP just now, seems to be working. Whoops, tried a second time, gone again! It’s down.

whoa: it’s now 10:05 CST and everything just came up.

can you stop looking into and just fix it?

Wait! I just got in to webmail.domain.com – but it was super slow. Just wanted to clarify.

I’m definitely looking into other solutions. I will post here after hearing from my tech consultant. I should have listened to him when he discouraged me from using Dreamhost.

October 9th, 2013 at 8:07 amDavid Kittredge Says:

Back up now!

I was just able to download my e-mails in Outlook but now I’m getting an error saying that my mail server cannot connect to my incoming pop3 e-mail server.

October 9th, 2013 at 8:08 amDewayne Adams Says:

Still having problems with IMAP!

I would thank an stimated fix time un the next update. Thanks

I’m so done with Dreamhost. Why does email go out this frequently? It’s not possible to run a business with this kind of service. It’s 10:10am Central Time and it’s been down since last night for me. Frustrating when I have deadlines. Unacceptable, really.

Cannot receive email

Some junk emails came through … back down again. Nothing stable. Still down and still not able to send projects out – Yikes ! ! ! ???????

I have 30 emails address without Access… :(

Hope the problem can be fixed son!! Thanks…

SMTP ok, webmail intermittent, POP3 not working

Update regarding “Forward-Only Addresses”
I’ve had one of my test emails show up. Three others remain AWOL.

Yep. SMTP was out all morning. Just started working again. No issues w/ IMAP today. That I know of.

my email working now

no mail access at all – through outlook, webmail or mobile.

WE ARE WORKING AGAIN – THANKS

10:18 (CT) Was able to send/receive … Thank You ! ! ! !

Having intermittent failures with deliverability and forwarding

still not working – status update or timing for the fix please?

Still having POP/IMAP issues. The backlog may be processing now, since messages from last night have now come in.

I think dreamhost is getting so much business from tiny start-up blogs, I don’t think they care anymore about their real business. Just look at the apathetic and evasive responses you get when you email customer service. We’ve had problems with mail for months and they kept saying there was no problem. They do the same thing with our web server. They just don’t want to deal with a problem unless it affects enough people.

Unfortunate really what they are turning into.

Hopefully this can be finally resolved. Mail has been flakey for a long time. Reported very similar (but intermittent) issues earlier in the week but was brushed under the carpet. Advice received earlier from DH support was to have our email hosted elsewhere. This makes me sad :(

All my emails down

SMTP IMAP is down , POP intermitent!

We now have intermittent emailing capabilities.

connects ok but doesnt recieve email, sends it thought

My email is working again. Seems fine. Thanks.

does as it was sent but is not recieved

My clients whole company is down? Did you learn how to run email systems back in the 90s? I don’t feel like there should be email breaks in 2013 or else this company has bigger problems it seems. Just fix it please, our clients have no idea what is going on, they blame us, and ultimately, that makes us need to move our business away from Dreamhost because it can cost us business too…

SMTP IMAP is down , POP intermitent!… All my emails down :(

sorry, did not explained it properly, tried sending it to an external email address, works as it was sent but it is not recieved by the external email address

October 9th, 2013 at 8:31 amLizzie Kaplan Says:

Still not receiving any emails today.

A quick test most of you can easily do (users of some more recent versions of Windows [7+] may need to install “Telnet Client” using the “Turn Windows Features On/Off” Control Panel, or by typing “pkgmgr /iu:”TelnetClient”” at the CMD prompt — this is a one-time thing and should not require a Windows restart) is to go to your operating system’s command prompt (CMD for Windows, Terminal for Mac OS X, etc.) and type:

telnet your.mail.server.dreamhost.com 110

The command prompt window should go blank, then, within a second or two at the most, you should see an identification message for the POP3 mail server and an instruction to log in. If you see this, then the mail server is running (or else you have antimalware software installed that scans incoming mail — such often works by installing a POP3/IMAP proxy server that intercepts all attempts to connect to POP3/IMAP and replaces them with its own server, which in turn connects to the real server you requested — the identificaton message should help you determine if this is the case, and if so, type “QUIT” to exit it, then temporarily disable your antimalware’s Email scanner [both incoming and outgoing], then try again).

What may happen instead is that you get an error message saying that it could not open a connection to the server, in which case check your typing of the server name. You can also use an alias using any of the domains you have set up for Dreamhost to handle Email for (pick a short, easy-to-type one), e.g.:

telnet mail.your.dh.domain.com 110

If you still get that message and you’re sure you typed the domain correctly, then that means that Port 110 (standard unencrypted POP3) is not open at all at the IP address that the fully qualified domain resolves to. This is NOT the problem that seems to be happening now.

What we’ve been getting here is that the command screen gets cleared as usual, but no identification message appears. The cursor just sits there at the top of the screen, flashing. This indicates that Port 110 is open, but either no listener is running, or the listener is not responding, or, as is more likely the case here, the machine in question is simply a proxy at DreamHost that forwards requests to the real in-house mail server and THOSE aren’t responding.

If you DO get the identification string, you can simply type “QUIT” to exit, or you may want to do more tests. To test actual POP3 functionality, find an Email address and its password of an account on that server, then type:

USER username@your.dh.domain.com (the FULL EMAIL address, NOT just the username!)

You should see a prompt, “username@your.dh.domain.com Ok. Enter password.” Type:

PASS p@$Sw0rD (of course, use the real password here — you can use your OS’s Copy/Paste feature if need be, but note that at least for Windows that pasting into a Command Prompt window often differs from the usual: for the standard Windows Command Prompt, you either click with the MIDDLE mouse button [often wheel button] if QuickEdit is enabled for that window, or use the window menu at the upper LEFT corner of the Command Prompt window, then go to “Edit,” then go to “Paste”)

You should see an acknowledgment of successful login (or maybe a password invalid message if you mistyped it). You can then type:

STAT

This will return the number of “unread” messages on the server, and the total storage they’re using in bytes. If you see this, then everything is working fine and you can QUIT.

My Apple Mail is working…all is good!

We have been having email issues for about 3 months now! Delayed emails, webmail issues.

Dreamhost told me that they cannot do anything about it until the hardware has been updated, and that the hardware won’t be updated for 3 months!

At the risk of jinxing things, my POP mail might be working. I sent a test to a DH account and haven’t received it yet probably due to lots of mail that has to be sent out so the queue is backed up, but I did immediately receive a test message I sent to a Gmail account.

Thanks.

And thanks again for posting this issue on the DreamHost Status page.

Sheila

Still getting intermittent errors.

My squirrel mail just stopped working :(

10:35AM CDT

Our email still isn’t working even after your update that states it should be ok: The following error occurs:

Task ‘SCFC – Sending’ reported error (0x800CCC0F) : ‘The connection to the server was interrupted. If this problem continues, contact your server administrator or Internet service provider (ISP).’

Thanks.

Keep in mind that DreamHost makes it very easy to use Google Mail for any account hosted at DreamHost. Especially for smaller clients and non-profits, this could be a very good alternative. The DreamHost Control Panel will save you the step needed of setting up the multiple MX-type DNS records that Google Mail uses, as well as the DNS SPF record. You still have to do the other steps.

Of course, even the mighty Google has suffered outages recently (including a system-wide one last month that caused traffic on the entire Internet to drop by 40% for about five minutes), so this is no absolute guarantee of uninterrupted service.

Email looks like it’s back up. Still having minor errors getting kicked off, but overall working.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Load Balancer Maintenance US-West Data Center (Los Angeles) – Redundant Power Upgrade – October 1st 8pm PDT (4 hours) (RESOLVED)
1:11 pm

Load Balancer Maintenance US-West Data Center (Los Angeles) – Redundant Power Upgrade – October 1st 8pm PDT (4 hours) (RESOLVED)

Posted (September 27th, 2013 at 1:11 pm PST) by Brian@dreamhost

On October 1st at 8pm PDT, we will be conducting maintenance on our Load Balancers in order to increase the power redundancy. The maintenance is scheduled to last 4 hours. We plan to move the hardware over in a series of monitored and pre-tested steps which should eliminate most service impact. Services that could be affected by this maintenance are listed below:

*All Mail Services – webmail – POP/IMAP – SMTP
DreamHost Web Panel

Our support team will be monitoring this status post and opening it up for comments once the maintenance starts. If you have any questions or concerns until then, please contact support.

*- Please note, even though the maintenance is in our US-West Data Center (Los Angeles), all customer Mail service and Panel service can possibly be affected. Mail sent from hosting machines, such as formmail responses, for example, will not be affected by this maintenance.

Update: October 1, 2013, 11:04 pm PDT: All done! Everything went according to plan and we’re resolving this maintenance! We thank you for your patience! If you are still seeing issues, or have any questions regarding this, please contact support and let us know so we can help!

Update: October 1, 2013, 10:23 pm PDT: Software update is now complete, we’re moving on to changing some network locations around. This can possible affect mail, specifically delivery, as well as checking and sending mail. We’ll update this as we move along.

Update: October 1, 2013, 10:10 pm PDT: Everything is prepped correct and now we’re upgrading several pieces of software. This should take about 20 minutes. We don’t anticipate any customer downtime due to this.

Update: October 1, 2013, 9:50 pm PDT: Phase 1 is now complete, we’re moving over to the 2nd half. We’re preparing now, and should be ready to go in a few moments.

Update: October 1, 2013, 9:44 pm PDT: We’re in a holding pattern right now, making sure everything is correct before moving on to the next step.

Update: October 1, 2013, 9:22 pm PDT: We fixed the issue with webmail, and all of our tests are working correctly. If you’re still seeing problems, please let us know.

Update: October 1, 2013, 9:15 pm PDT: Some users are still reporting problems with mail service, specifically when using webmail. We’re still investigating that and should have more info soon!

Update: October 1, 2013, 8:57 pm PDT: One of the load balancers we’re updating took a little longer than expected to re-intitilize. This is caused some mail service problems. These should be resolved now.

Update: October 1, 2013, 8:41 pm PDT: Everything is going as expected, we had a small (a few seconds) interruption in mail service, however, that’s now been resolved. We’re still moving over some of the load balancers that handle mail service so this could continue for a bit.

Update: October 1, 2013, 8:15 pm PDT: That went well, so we’re moving on to our load balancers that handle the web panel, webmail and some internal traffic. We don’t expect downtime to result from this.

Update: October 1, 2013, 8:06 pm PDT: We’re beginning now, however are only switching internal services over at this time. This will not result in any customer facing downtime.

Update: October 1, 2013, 7:55 pm PDT: We’re here and getting set up to start. We’ll let update when we are starting.

This entry was posted on Friday, September 27th, 2013 at 1:11 pm and is filed under Data Center Maintenance, Email Issue, General Outages, Major Outage, Multiple Server Issue, Network Maintenance, Network Outages, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

25 Responses to “Load Balancer Maintenance US-West Data Center (Los Angeles) – Redundant Power Upgrade – October 1st 8pm PDT (4 hours) (RESOLVED)”

Mail is offline. 1.40pm AEST.

@scott. Sorry about that! We did have a bit of a blip on mail service, however, that should be fixed! If you’re still seeing problems, please let me know what errors you’re seeing. It might be better to submit a support ticket with specifics, you can make the subject “MAIL OFF LINE, BRIAN H” and I’ll have someone check it for you.

Mail has been down for at least 10 minutes. It is now 11:48PM EST.
The error is-

The connection was reset

The connection to the server was reset while the page was loading.

squirrelmail says:

ERROR:
ERROR: Connection dropped by IMAP server.

@jen and @paul can you let us know what domain names you’re having problems with?

pxnx.us

Never mind! Seems to be back up now. Thanks!

My mail is down too (enlighted.com, on homiemail-sub5)

@Janet, we’re looking into it for you!

Sorry, spoke too soon. Now squirrelmail sez

ERROR:
ERROR: Connection dropped by IMAP server.

again
pxnx.us

@paul, thanks, we’re investigating

I have several accounts at Dreamhost and webmail is still giving me “ERROR: Connection dropped by IMAP server.” on all my accounts.

@Paul, @Janet, @Jen Webmail issues should be fixed, can you verify you’re all good for me? sorry about that!

@Gerry, we just fixed that, can you check again and let us know if things are still broken?

I’m (still) seeing the ‘connection dropped’ and am timing out on several of my domains webmail. :(

@Joe can you let us know what address you’re testing on?

sure im using blankslatejoe@terribleposture.com as a primary testing one. Lots of time-outs

My mail works now.

It is 12:30AM EST.

Thanks.

Oh! It’s speedy now! Whatever you did, thanks!

Getting spotty service and “ERROR: Connection dropped by IMAP server.” messages on homiemail-sub4 and homiemail-sub3.

Hey there, My website is down, is because maintance or is happening anything else with my server?

Oh, whatever you did, it’s speedy now! Thanks!

Oh! Seems to be OK now.

Awesome. @Bruno, I tested the site you linked to, and it works, are you having problems with a different domain? Web traffic wouldn’t be affected by this.

@Brian I tested right now and works. No all domains that I have are redirecting to the link that I linked to. So tks because now is working.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Monday September 23rd 10am – 1am PDT
11:21 pm

Live Chat Unavailable Monday September 23rd 10am – 1am PDT

Posted (September 20th, 2013 at 11:21 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Monday, September 23rd, from 10am to 1am PDT (the following morning). We will be involving our support staff in a special, employee appreciation day, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 1am PDT the next morning, September 24th. We thank you for your patience and understanding.

This entry was posted on Friday, September 20th, 2013 at 11:21 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Server configuration issue impacting a small subset of VPS servers
12:35 am

Server configuration issue impacting a small subset of VPS servers

Posted (September 17th, 2013 at 12:35 am PST) by mir

We are currently investigating an issue with one or our network configuration scripts which is disrupting networking for VPS guest servers on a number of VPS host machines. No other services (shared hosting/email/database/panel/etc.) are affected and we are in the process of fixing all host machines and their guests while also looking into the root case so we can also correct that. Currently under one tenth of customer VPS guests (and the sites on them) have been disrupted. We apologize for the inconvenience and will provide updates as they become available.

Update: September 17, 2013, 2:55 AM PDT: The clean up is finished and everything looks good. We are calling this resolved. Please contact us (Support -> Contact Support in the admin panel) if you are still experiencing any issues on your VPS. Thank you for your patience!

Update: September 17, 2013, 2:16 AM PDT: All reconfiguration commands have completed! We are now going through everything to make sure there are no lingering issues. We are tentatively calling this resolved.

Update: September 17, 2013, 2:12 AM PDT: We have processed over 92% of the reconfiguration commands and are estimating completion in under an hour. We’ll update this post as we make progress.

Update: September 17, 2013, 1:48 AM PDT: We have processed over 43% of the reconfiguration commands and are estimating completion in under an hour. We’ll update this post as we make progress.

Update: September 17, 2013, 1:38 AM PDT: We have processed over 37% of the reconfiguration commands and are estimating completion in under an hour. We’ll update this post as we make progress.

Update: September 17, 2013, 1:28 AM PDT: We have processed over 25% of the reconfiguration commands and are estimating completion in an around an hour (throughput has increased lowering the expected time to finish). We’ll update this post as we make progress.

Update: September 17, 2013, 1:20 AM PDT: We have processed approximately 15% of the reconfiguration commands and are estimating completion in an hour or two. We’ll be sure to keep everyone posted until the matter is completely resolved.

Update: September 17, 2013, 1:00 AM PDT: In order to correct this issue we’re having to run a configuration for all of the impacted VPS machines which have to be scheduled carefully (we don’t want to overload the system that manages updates). These are in the process of running and the fix is working; however, it will take some time to run for every VPS guest that needs it. We’ll continue to update this post as we make progress.

Update: September 17, 2013, 12:45 AM PDT: Upon further investigation we’ve found that the number of affected VPS guest servers is significantly lower than initially thought (less than 10%) which should speed the cleanup process. We apologize again for the disruption and thank you for your patience!

Update: September 17, 2013, 12:39 AM PDT: We have determined the root cause and are in the process of implementing a fix. We’ll provide an ETA as soon as we’re able to determine how long it will take to complete the repair.

This entry was posted on Tuesday, September 17th, 2013 at 12:35 am and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.6.1 Security and Maintenance Release (RESOLVED)
3:04 pm

WordPress 3.6.1 Security and Maintenance Release (RESOLVED)

Posted (September 11th, 2013 at 3:04 pm PST) by Mika E.

WordPress 3.6.1 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Unless disabled, automatic upgrades for DreamPress and other One-Click installs will begin shortly (of course, you would never disable them, right?). If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.

This entry was posted on Wednesday, September 11th, 2013 at 3:04 pm and is filed under Announcement, DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “WordPress 3.6.1 Security and Maintenance Release (RESOLVED)”

500 Internal Server error on most of my sites right now.

At least 15 of my sites are unusable or unbearably slow at best.

Both VPSes completely dead right now, no ping, nothing. Lovely….

I am having the same problem since upgrading to wp 3.6.1. Sloooowness. 30 seconds or more of “waiting”.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency email hardware maintenance on homiemail-sub4 cluster – RESOLVED
1:57 pm

Emergency email hardware maintenance on homiemail-sub4 cluster – RESOLVED

Posted (September 11th, 2013 at 1:57 pm PST) by andrea

Due to the issues we had with our email service yesterday and today on our homiemail-sub4 cluster, we feel it’s in our customers’ best interest to go ahead with a network maintenance. This is to alleviate the problems that have been popping up yesterday and today. Our System Engineering and Network Engineering teams have staged and tested the process of swapping out the hardware to a more robust switching infrastructure, and we aren’t expecting any customer-facing downtime. The process itself will take less than a minute. We are posting this to let you know that we’re working toward further stabilizing our email services. We appreciate your patience and apologize for all inconvenience this has been causing.

This maintenance is related only to email services on said cluster. To check if you’re affected, please go to your DreamHost panel. If you have any questions or concerns about this or anything else, please contact support with the details.

Update: September 11, 2013, 2:49 PM PDT: The maintenance was completed successfully. If you are still seeing issues, or have any questions regarding this, please contact support and let us know so we can help!

This entry was posted on Wednesday, September 11th, 2013 at 1:57 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Emergency email hardware maintenance on homiemail-sub4 cluster – RESOLVED”

How do I tell if I’m on sub4? It says homiemail-master and I have been having tons off email issues the last few days. I have contacted support and am waiting to hear back.

@scotepi: If you are homiemail-master, you aren’t on sub4, so please make sure you send in a support request (http://dhurl.org/dhsupport) and ask us about your issues (if you haven’t already!)

Also… This notice is just now popping up today, shouldn’t it have been posted yesterday to acknowledge the issue to start with? What is the point of DH Status if you wait until the problem is fixed before posting about it? BTW – fixed or not it still seems like there are issues?

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency Network Maintenance US-West Data Center (Los Angeles) Friday September 6th 9pm PDT (6 hours)(RESOLVED)
7:56 pm

Emergency Network Maintenance US-West Data Center (Los Angeles) Friday September 6th 9pm PDT (6 hours)(RESOLVED)

Posted (September 6th, 2013 at 7:56 pm PST) by Brian@dreamhost

Due to the network problems outlined here we are having to perform emergency maintenance tonight at 9pm PDT. This maintenance should last approximately six hours

This procedure will move more networks over to our new hardware, very similar to what we did here last thursday night, however, because of the emergency nature of this procedure, there are several larger differences.

-First, we anticipate some mail service interruption. This includes mail delivery, IMAP, POP and SMTP as well as webmail and mailboxes.

-Second, this will also affect core services like https://panel.dreamhost.com.

We apologize for the very short notice, and possible disruptive nature of this maintenance, however, we feel it’s absolutely needed in order to stabilize the network service we provide our customers. As always, we will be providing details the maintenance progresses

Update September 7th 1:50PM PDT: We have cleaned up any remaining issues and we are resolving this maintenance.

Update September 7th 1:35PM PDT: DNS issues are now resolved and we are checking any new support requests to check if there are any further issues. We are near calling this maintenance clear as our tests are showing websites, mail, and panel working correctly.

Update September 7th 1:20PM PDT: Most services are restored now. We are still working on some lingering DNS issues. The majority of the work is done, we are in the clean up phase right now.

Update September 7th 12:30PM PDT: We have our network set up the way it is designed to be again. We are now troubleshooting any possible lingering issues and making sure that the issue is resolved. We are not out of the woods just yet but we are getting closer.

Update September 6th 11:55PM PDT: Reboot is complete but we now have to move services back again. Service should be restoring as we progress. We are also mobilized to perform manual reboots of machines having issues as services return.

Update September 6th 11:50PM PDT: We are now rebooting some of our network hardware. This will cause disruption to mail related services and to our panel as well. We thank you for your patience during this service interruption.

Update September 6th 11:40PM PDT: Some of our moves have caused DreamHost.com to go down. This is expected as part of this emergency maintenance. We will continue moving forward with this maintenance and we are close to the next step which will cause the mail and panel interruption listed in the main announcement above.

Update September 6th 11:15PM PDT: We are still in the process of methodically moving each networks carefully. We test each moved vlan after it is complete. We are still working on non-customer facing moves.

Update September 6th 10:38PM PDT: Ok, that network moved over properly, with no downtime! Moving on to more internal, non-customer facing networks now, so no downtime should occur.

Update September 6th 10:00PM PDT: At this time, we’re about to move a network that has the potential to cause some downtime. Most likely web and mysql traffic could be affected, for a brief period of time.

Update September 6th 9:45PM PDT: We’re now moving a small range of the network, however these networks are not customer facing.

Update September 6th 9:03PM PDT: We’re here, and setting up now. We’ll post when our first move is about to get underway, and let you know what to expect.

This entry was posted on Friday, September 6th, 2013 at 7:56 pm and is filed under Announcement, Control Panel Outages, Email Issue, General Outages, Network Maintenance, Network Outages, Webmail Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

19 Responses to “Emergency Network Maintenance US-West Data Center (Los Angeles) Friday September 6th 9pm PDT (6 hours)(RESOLVED)”

email service is down in all my accounts

ok boys, i have read above. web and ftp works now for me. Waiting now for email and panel access

web and ftp are working for me, but not ssh access.

@ionisse – we are right in the middle of access being down, I will update as soon as everything should be working. @Paul – our network is in a bit of a vulnerable point until we can get this part of the maintenance complete. Please hang in there and I’ll update the post when things should be working. If you get my all clear and still have issues, please make sure to comment with your domain so I can get it tested for you.

all my mails are down….

@Mayank Agrawal – This is expected. Mail service should be restored soon. If it does not come up we have our operations team ready to work on any outstanding issues. Once I give the clear in the status post, please reply if there are still issues. Thanks!

When you get a chance please just tell us if it was a squirrel, russian hackers, PRISM, spilled pickle juice, etc. At least for the story aspect of this whole event.

@joel – emergency maintenance was related to the network maintenance we did on Thursday. We encountered an unexpected bug, which required the reboot we are performing right now. Sorry the story isn’t juicier.

I’ve only just noticed one of my sites is down.
CloudFlare gives a 1001 DNS Resolution Error, while all ajax requests return a 409 Conflict Error.
My other sites, not using CloudFlare, are all up and running!

@sidepa – Should be resolved soon. There were some temporary DNS issues that likely got cached with cloud flare during this maintenance. Apologies. I’ll keep an eye on your domain.

email service working fine now, boys. Thank you for your job.

I’m not using cloudflare. my webs are running. Just expecting for panel

Neither http://einarpetersen.com nor http://globability.org are reachable ????

Are you getting closer to getting out of the woods yet ???

And yeah I know this is not the right forum but…

Your site will not process the Submit – at least nothing happens when pushing the send message now button.

As a network admin I’d like a technical description regarding the bug that has plagued you.

@Einar – where is that problem exactly? Unfortunately we can not release the specific technical issue at this time.

Web, SSH, FTP still not working, already 9 hours

@Riza Hisamuddin – checking in to this. Does not appear to be related as your site is in a different data center than the maintenance was performed in. Could be DNS, however your actual ps is having issues, not the machine the ps is located on.

webmail.nathanson.org is still down. It has the DH 1-click of Roundcube.
http://www.nathanson.org is up. Thanks for all you guys do.
mail services for my domain seem ok now.

@Riza Hisamuddin – issue was unrelated, I restarted your vps and it’s working now. the team also noticed that you are using over the resources you have set in the panel. You can contact our support team to help you troubleshoot what is using your vps resources.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues at US-West (Los Angeles) Data Center (RESOLVED)
3:04 pm

Connectivity issues at US-West (Los Angeles) Data Center (RESOLVED)

Posted (September 6th, 2013 at 3:04 pm PST) by Sean

At this time, we are currently experiencing connectivity issues at our US-West (Los Angeles) Data center. Our teams are investigating the root cause and are working towards a resolution. We’re very sorry for the inconvenience this is causing, and will post updates as the work progresses. Web panel access, webmail and mailboxes (in addition to web, mysql and email) may be affected by these problems.

To see if you may be affected, please check your location in the DreamHost panel.

Please follow here for updates, and thank you for your patience!

Update August 6th 8:50PM PDT: We’re resolving this status post for now, and rolling these problems into the maintenance coming up tonight at 9pm PDT on friday, september 6th 2013. You can view that post here

Update August 6th 5:59PM PDT: Services are still stable and our admins are still working with our hardware vendor to find a root cause, as well as a resolution. We’ll keep you all posted as more information becomes available.

Update August 6th 5:04PM PDT: Services are still stable for the most part, and our admins are far along into the necessary investigations and maintenance work with an engineer from our hardware vendor. We’ll be updating this post again as more information becomes available.

Update August 6th 4:04PM PDT: Services are relatively stable for the moment with the occasional quick hiccup. Investigations are ongoing and we will be updating this post with more information as soon as we have it. Your patience is appreciated!

Update August 6th 3:40PM PDT: We’re still seeing intermittent service interruptions, and our team is working alongside our network hardware vendor to determine the root cause. We apologize for any inconveniences caused and will update this post again as soon as we have more information.

Update August 6th 3:21PM PDT: We have not experienced and interruption of service in the past 20 minutes.  Our team is still investigating the issue further and we are in touch with our network hardware vendor for further insights on the possible cause of these multiple, quick network interruptions.

This entry was posted on Friday, September 6th, 2013 at 3:04 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Google Authenticator Update for iOS Loses Account Information
11:34 pm

Google Authenticator Update for iOS Loses Account Information

Posted (September 3rd, 2013 at 11:34 pm PST) by Steven

Word on the internet is that the latest update to the iOS Google Authenticator app may lose account information on upgrade. If you have it set up on your DH account, this will lock you out of the admin panel (https://panel.dreamhost.com)! This is, obviously, bad. We recommend against upgrading the app (on iOS only) for the moment.

This entry was posted on Tuesday, September 3rd, 2013 at 11:34 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

5 Responses to “Google Authenticator Update for iOS Loses Account Information”

A lesson to app developers in regards to iOS 7, it auto updates by default. Fix these bugs BEFORE pushing to Apple.

Whew … one more reason for me to be grateful I’ve stuck with Android.

Great advice not to update, but what if we already have? How do we re-activate?

@Keith, I would imagine if DH has something set up for if you loss your phone, you’ll have to use that.

It looks like Google has pulled it from the Apple app store: http://www.tuaw.com/2013/09/04/google-releases-pulls-authenticator-app-version-20/

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West Data Center (Los Angeles) Thursday September 5th 8pm PDT (6 hours) (RESOLVED)
9:31 am

Network Maintenance US-West Data Center (Los Angeles) Thursday September 5th 8pm PDT (6 hours) (RESOLVED)

Posted (September 3rd, 2013 at 9:31 am PST) by Ralph

On Thursday, September 5th at 8pm PDT, we will be replacing our core network hardware in our Los Angeles US-West data center.

The maintenance window is being set at 6 hours to allow us to move methodically so we can limit any potential service degradation. The process will be complete in a series of steps.
Our initial step will require a reboot of our border hardware. During this step there will be degraded service for a period that is expect to last under 30 minutes, however sites will be up and working during that time.

Once that is confirmed complete our new hardware will be connected to the border and we will carefully move subsets of our network from the existing core, to the new core. In our previous maintenance we have been able to accomplish this with minimal to no impact on site availability, barring any unforeseen issues. If we do come across a problem we can easily roll the change back to previous core.

When all virtual networks are moved to the new core we will have completed all major network hardware replacements for the US-West Data Center (Los Angeles). Comments will be open and monitored during this maintenance so that we can work with you and assist with any issues that may come up during the maintenance.

As always, please contact support if you have any questions. You can also check if you may be affected by this in your DreamHost Web Panel by checking to see if you have services in the US-West data center (Los Angeles).

Update – September 6th, 12:20AM PDT: We’re done! Everything went according to plan and we’re resolving this maintenance just shy of 2 hours early! If you’re seeing problems with your site, please contact support asap!

Update – September 5th, 11:02PM PDT:At this time, we are more the 50% through our scheduled maintenance and anticipate an on time completion.

Update – September 5th, 10:35PM PDT:Everything is going smoothly, we’re updating those network subsets slowly but surely, to minimize downtime. Currently, we have seen no issues pop up due to these moves!

Update – September 5th, 10:00PM PDT:According to preliminary tests, that move went well. We’re monitoring right now to make sure things are working as expected. We’ll be moving smaller subsets in the next few minutes. For those, possible downtime is the same, in the seconds to minutes range.

Update – September 5th, 9:45PM PDT:We’re moving a larger subset of network IP ranges in a few minutes. This could affect web, mysql and mail for a larger subset of users (than we were previously moving) in this data center. Any periods of inaccessibility should be within the seconds to minutes range.

Update – September 5th, 9:10PM PDT: Alright, we’re about to start moving subsets of our networks over to new hardware. This part could result in some downtime, however, that should be brief! We’ll keep updating as it progresses.

Update – September 5th, 8:30PM PDT: We are still moving forward, all updates at this time are not customer facing and we do not expect any downtime to result from them.

Update – September 5th, 8:16PM PDT: Ok, we’re ready to start. Our first step is to enable a few settings in our network hardware. You should see no impact due to these changes, and we expect no downtime to result.

Update – September 5th, 7:55PM PDT: We’re all here getting set up to start. We’ll update this when we get situated and are ready to begin.

This entry was posted on Tuesday, September 3rd, 2013 at 9:31 am and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail connectivity issues affecting homiemail-sub3, homiemail-sub4, homiemail-master and webmail services (RESOLVED)
3:55 am

Mail connectivity issues affecting homiemail-sub3, homiemail-sub4, homiemail-master and webmail services (RESOLVED)

Posted (August 30th, 2013 at 3:55 am PST) by chalise

We are currently investigating stability issues on the machines that handle mail service on three of our mail clusters, as well as webmail which is causing connection errors when trying to send or receive.

You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the “Account Status” tab. Scroll to the bottom and you will see a field labeled “Your Email Cluster”.

We apologize for the inconvenience and greatly appreciate your patience while our team works to resolve this.

Update – August 30th, 4:50AM PDT: This is now resolved and all mail servers and webmail services are working normally once again. If you’re noticing any remaining mail issues, they are not related, so please be sure to contact support under “Support” / “Contact Support” in your panel. Thanks again for your patience!

Update – August 30th, 4:30AM PDT: We are seeing definite improvements in services across all clusters, and multiple test cases are working as well. We are still continuing to work on resolving this, and will provide more detail as we have it.


This entry was posted on Friday, August 30th, 2013 at 3:55 am and is filed under Email Issue, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

18 Responses to “Mail connectivity issues affecting homiemail-sub3, homiemail-sub4, homiemail-master and webmail services (RESOLVED)”

Mail not responsing

mail not resonsing

I’m not able to send or receive and am on homie master so hope it’s back up soon.

Why does this post not show up on the main index page? (Only found it via a link in the Panel that just appeared.)

Just wondering how long would it takes?

Mork: IMAP, POP & SMTP are all down (along with Webmail) – you won’t be able to send or receive until this outage is resolved if you’re on one of the affected clusters.

I’m having trouble too, but my cluster is listed as ‘homiemail-sub5′. So the error might be wider than: homiemail-sub3, homiemail-sub4, homiemail-master
Or the 3 mentioned are affecting other clusters receiving mail.

Ow boy; every time it goes down it’s in the USA night; but as EU customer it’s getting pretty frstrating to not being able to use mail as I should be _during the day_…

My serwer is affected too. Please check it

Way doesn’t this post appear on the main page at dreamhoststatus.com? The only way I found it was by logging into the panel and following the link. Nothing on twitter, either.

Any word on this effecting site servers? All of my sites have a Gateway Time Out error, that plus my mail being down literally renders me useless. Submitted ticket an hour or two ago and nothing yet. I usually dont make comments complaining about something but there has been a ton of downtime for me lately. If I wasnt hosting 29 sites with you I would consider looking elsewhere but dont feel like possibly having 29 problems right now.

Please fix comments – they only appear by appending “comment-page-1″ to the URL. Otherwise it looks like users aren’t replying.

we are receiving and sending properly. thanks

Why are comments not appearing?

The emails sent will be lost during the incident?

mail is working again :)

Gentlemen, good morning. I pointed Dreamhost for my boss, because I’ve used it before and found it very good. But lately there is much problem delay delivery of emails and system instability. Because of this my boss is wanting to unsubscribe. A hug.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Live Chat Unavailable Wednesday August 28th 9am – 10pm PDT
4:38 pm

Live Chat Unavailable Wednesday August 28th 9am – 10pm PDT

Posted (August 26th, 2013 at 4:38 pm PST) by Ralph

Our live chat support service will be unavailable this coming Wednesday August 28th from 9am to 10pm PDT. We will be having a company meeting which will require our live chat team to be present. Email support operations will still be available. We will resume normal Live Chat operations at 10pm PDT if not earlier. We thank you for your patience. We will aim to get live chat up and running as soon as we can once the company meeting is complete.

This entry was posted on Monday, August 26th, 2013 at 4:38 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » One Click Installs/Upgrades Temporarily Offline (RESOLVED)
8:32 am

One Click Installs/Upgrades Temporarily Offline (RESOLVED)

Posted (August 6th, 2013 at 8:32 am PST) by Oscar

Due to a hardware issue with one of the servers which processes the installations/updates of our one click installs, we’ve had to temporarily disable the installation and upgrades for all OCI’s. This will not affect your OCI if it was previously installed, it’s only affecting any new or pending upgrade OCI’s. We are working to restore all OCI functionality and we will update this post as soon as we have more information.

Update August 6th 2:10PM PDT: Great news! All one click installs and upgrades are now being processed again. There will be a delay before your one click install and upgrades are processed, but you will receive an email from us when complete. We sincerely apologize for any inconvenience that this delay has caused you. Should you have any issues with your OCI’s, please contact our support team and we will be glad to look into the issue for you.

Update August 6th 12:30PM PDT: We are still trying to resolve this issue and get all delayed installs/upgrades back online. Please note, this is not affecting any other services or one click installs that have already been installed or upgraded. This is only affected installations or upgrades that have been requested but have not yet been processed. This is not causing any downtime on your site either. Again it’s only the software that is pending install or upgrade that has yet to be processed. We will update this post again as more have more information.

 

And this is resolved! You should not be seeing any problems with connection at this point. If you are having any issues with your sites, please contact support, so we can assist you.

 

This entry was posted on Tuesday, August 6th, 2013 at 8:32 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

16 Responses to “One Click Installs/Upgrades Temporarily Offline (RESOLVED)”

Actually every site I have is down with SQL errors, so the outage may be bigger then you think. Ticket submitted.

oh my god,all sites are down..

Yup, seems that the MYSQL servers are all down for the count right now.
I’m sure they will be back soon though :-)

Well I’m glad to see that there’s anything posted here at all. Sometimes when there’s clearly a server issue, nothing gets posted to this page.

But isn’t it possible to tell us more clearly where the outage is happening so that we don’t all waste our time (and yours) filing individual tickets?

My site was briefly down, then back up. I made a post and informed many people that it was up. Now the site is down again.

My site (just one of them) is down as well — just got a 503 a few minutes ago, while trying to do some site maintenance…

I have 12 hours outage with homie-vserver84. There is now status page and there are no updates for hours. All my sites are down. Dreamhost serveice is awfull.

August 6th, 2013 at 11:05 amAndy Ringsmuth Says:

homiemail-sub3 e-mail is all f’d up again too. It must be $day.

My email is down.

It seems some sites are active and others, not so good…

My email is intermittent at best. Hard to do business this way.

Email has been very slow to come through today for us as well. Very frustrating.

I know this is offtopic but since there is no ontopic thread :

also experiencing very slow mail delivery. Incoming is delayed by roughly an hour. No problem sending. I wish Dreamhost would use their own status page occasionally.

August 6th, 2013 at 11:48 amTim Golightly Says:

Seems that dreamhost are extremely unreliable, their server at 75.119.215.148 (ps6845.dreamhost.com) has been down for almost 24 hours now, probably best to move your sites to another host.

Our site is giving a 503 error as well since this morning

August 6th, 2013 at 11:53 amTim Golightly Says:

Also, you should demand refunds for any service downtime, e.g if you pay $10 per month and your site is down for a day then a refund of 0.33c is required.

Im guessing the panel login is jacked too

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues at US-West and US-East Data Centers (RESOLVED)
2:12 pm

Connectivity issues at US-West and US-East Data Centers (RESOLVED)

Posted (August 3rd, 2013 at 2:12 pm PST) by Elizabeth

We are seeing connectivity issues at US-West Data Centers (Los Angeles and Irvine) and our US-East Data Center. Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses. To see if you may be affected, please check your location in the DreamHost panel.

We’re sorry for any issues and thank you for your patience!

Update August 3rd 2:45PM PDT: And this is resolved! You should not be seeing any problems with connection at this point. If you are having any issues with your sites, please contact support, so we can assist you.

This entry was posted on Saturday, August 3rd, 2013 at 2:12 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.6 Major Release Now Available (RESOLVED)
4:00 pm

WordPress 3.6 Major Release Now Available (RESOLVED)

Posted (August 1st, 2013 at 4:00 pm PST) by Mika E.

Rabbit, Rabbit!

The newest version of WordPress 3.6 (Oscar), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

3.6 is both a feature and maintenance release, and is highly recommended for all customers. Its new features include built in playback for audio and video, a bold new default theme, improved autosave and more!

For more details, you can check out the release notes here, with further details available in the WordPress codex.

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.6 in the DreamHost forums, and the 3.6 master list at the WordPress.org forums, where issues relating to upgrades are being tracked and updated as they come up.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

You may want to take this time to also clean up your site’s themes and plugins. Delete anything you’re not using, and upgrade everything else. And don’t forget to make a backup!

You should really go watch the video. It’s pretty cool.

This entry was posted on Thursday, August 1st, 2013 at 4:00 pm and is filed under Announcement, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “WordPress 3.6 Major Release Now Available (RESOLVED)”

My site is currently down.

Site is on coleus
SQL is on tuba.

Any known issues?

Hi there,

We’re experiencing a variety of issues with various Dreamhost sites, particularly regarding the following:

1. Media uploader doesn’t work, just spits a “Failed to write file to disk” error.
2. PHPMyAdmin doesn’t seem to be working consistently – it’s not exporting our database.

Any update on what’s going on?

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency Hardware Maintenance US-East (Virginia) Data Center, Thursday Aug 1st 8pm PDT (4hrs) (RESOLVED)
5:30 pm

Emergency Hardware Maintenance US-East (Virginia) Data Center, Thursday Aug 1st 8pm PDT (4hrs) (RESOLVED)

Posted (July 31st, 2013 at 5:30 pm PST) by andrea

We’re scheduling a vendor-suggested emergency maintenance for Thursday evening, 8pm PDT. The maintenance itself will be rolling reboots of redundantly set up hardware. Because of this set up, we don’t estimate any customer-facing downtime, but our teams will be closely monitoring the process, and we also have a high-level support technician standing by on the vendor’s end too. We estimate the process to take no longer than two hours, but we’re setting a four-hour window to run all necessary tests and confirmations.

We apologize for the late notice, but we feel that this will greatly improve performance without affecting any services in the process. If you have any questions, please don’t hesitate to contact support. We’ll update this status post as soon as the maintenance starts and will be posting updates throughout.

Update August 1st 8:02pm PDT: We are ready to proceed with the maintenance. We’ll be posting updates as we progress.

Update August 1st 8:31pm PDT: So far so good; the maintenance is going smoothly. We’ll report back again as we go through the stages.

Update August 1st 9:11pm PDT: We are happy to report that we finished way ahead of schedule and without any impact on our services. The maintenance is now successfully completed! If you are having any sort of issues with any of your hosting services, it’s unrelated to this maintenance, so please contact support with the details, and we’ll be happy to assist.

This entry was posted on Wednesday, July 31st, 2013 at 5:30 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Secure certificate changes coming for SMTP on all mail clusters on June 27th 2:30pm PST (RESOLVED)
3:45 pm

Secure certificate changes coming for SMTP on all mail clusters on June 27th 2:30pm PST (RESOLVED)

Posted (June 26th, 2013 at 3:45 pm PST) by andrea

We are happy to report that we are in the final stages of changing the secure certificates for all mail services for all our email clusters. A while ago, we rolled out the changes to IMAP and POP, and tomorrow, on the 27th, we’ll be doing the same for SMTP.

If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, here is a quick notice for you to prepare: you will likely run into a warning when you log into your email client again (after the change!), stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that. In some clients, you may have to drop the old certificate and restart the client then accept the new one.

Unfortunately, we can’t anticipate all email client behavior, but we would be happy to help, so please contact support if you need help sorting this out after we made the change.

Update July 2nd 11:11am PDT: The changes have been applied last week, and we’ve been monitoring them to make sure everything went well. Please contact support if you’re running into any issues with your mail SSL certificate, and we’ll be happy to help.

This entry was posted on Wednesday, June 26th, 2013 at 3:45 pm and is filed under Announcement, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.5.2 Security and Maintenance Release (RESOLVED)
1:39 pm

WordPress 3.5.2 Security and Maintenance Release (RESOLVED)

Posted (June 21st, 2013 at 1:39 pm PST) by Mike S

WordPress 3.5.2 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Unless disabled, automatic upgrades for DreamPress and other One-Click installs will begin shortly (of course, you would never disable them, right?). If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.

Update July 3rd 12:56pm PDT: This is finished as of Monday, July 1st, afternoon. If your install did not upgrade, or you run into further issues, please contact Tech Support via the DreamHost panel for assistance!

Update June 22nd 12:45am PDT: Automatic upgrades are running! If you see issues following your upgrade, please contact technical support for assistance.

This entry was posted on Friday, June 21st, 2013 at 1:39 pm and is filed under Announcement, DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “WordPress 3.5.2 Security and Maintenance Release (RESOLVED)”

This auto-updated deleted my entire webpage! I was developing it for a presentation for the steering committee of my nonprofit tomorrow, and now I have nothing. There were approximately 22 pages, profiles created, blog posts. What happened? Can I undo this? The Dreamhost restore didn’t work.

These updates are killing ALL of my WP sites. Support responded with “restore your domain and your database”. That doesn’t work. Is this a DH problem or WP problem?

Turn auto update off

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-East Data Center (Virginia) Data Center Friday June 21st 10pm PDT (4 hours) (RESOLVED)
4:46 pm

Network Maintenance US-East Data Center (Virginia) Data Center Friday June 21st 10pm PDT (4 hours) (RESOLVED)

Posted (June 19th, 2013 at 4:46 pm PST) by Brian@dreamhost

We are conducting network maintenance in our US East Data Center (Virginia) on Friday, June 21st at 8pm PDT. This maintenance involves moving networking routes to brand new equipment, while directing traffic through thoroughly pre-tested and staged backups.

The maintenance is set for a four hour window, and we estimate most of this to not be customer-facing.
To see if you may be affected, please check your data center location for mail, web and mysql services in your DreamHost panel.

We will post updates as the maintenance progress and will be monitoring comments. If you have any questions or concerns, please contact support with the details, so we can assist you.

We appreciate your continued patience while we improve our networking services.

Update June 22nd 01:13am PDT: We’re all done and are resolving this status post as we have successfully completed the maintenance. Thank you for your patience throughout this maintenance!

Update June 22nd 12:30am PDT: Phase 3 is still ongoing and there has been no impact on services thus far.

Update June 22nd 12:00am PDT: We are still in Phase 3 and things are still going smoothly. We’ll update this post again shortly!

Update June 21th 11:30pm PDT: We are still in Phase 3, but everything is going as expected. We will keep updating this post as we progress!

Update June 21th 11:00pm PDT: Phase 3 of this maintenance is going smoothly. We’ll continue to provide you with progress updates as we go along. We appreciate your patience.

Update June 21th 10:29pm PDT: Phase 2 has completed with no impact to services and we are moving onto Phase 3.

Update June 21th 10:18pm PDT: Phase 1 has completed without incident and we are moving onto Phase 2.

Update June 21th 10:04pm PDT: Phase 1 is now underway! As stated, our plan is to use the redundancy in our network to perform this maintenance with 0 impact of service. We will update this status post as we progress through the 3 phases.

Update June 21th 9:53pm PDT: Our team is ready for maintenance action! This will be a 3 stage process and we will be starting phase 1 very shortly. We do not expect any service interruption, but comments are now on and we will be evaluating them. We’ll update when things get underway!

This entry was posted on Wednesday, June 19th, 2013 at 4:46 pm and is filed under Announcement, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West Data Center (Los Angeles, CA) Wednesday June 19th 10pm PDT (4 hours) (RESOLVED)
12:33 pm

Network Maintenance US-West Data Center (Los Angeles, CA) Wednesday June 19th 10pm PDT (4 hours) (RESOLVED)

Posted (June 19th, 2013 at 12:33 pm PST) by Brian@dreamhost

Tonight, Wednesday June 19th, we will be performing maintenance on the network at our US-West Data Center (Los Angeles, CA). We will be starting at 10pm PDT with a 4 hour window. During the maintenance we will be restarting the links between our border and core routers, as well as our link connecting our data centers. The redundant network should give us the ability to perform these changes without impacting your service. Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

Update June 19th 11:06pm: Our network remains stable, and the maintenance is now complete. We appreciate your continued patience!

Update June 19th 10:52pm: We’re monitoring to ensure stability and the maintenance continues to run smoothly. Please check back for further updates.

Update June 19th 10:26pm: The maintenance is going smoothly. We’ll continue to provide you with progress updates as we go along. We appreciate your patience.

Update June 19th 10:07pm: We’ve begun the maintenance now and will provide you with updates as it progresses. Again, this shouldn’t cause any service interruptions.

This entry was posted on Wednesday, June 19th, 2013 at 12:33 pm and is filed under Announcement, Data Center Maintenance, Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail connectivity issues affecting homiemail-sub3 and homiemail-sub4 (RESOLVED)
10:23 am

Mail connectivity issues affecting homiemail-sub3 and homiemail-sub4 (RESOLVED)

Posted (June 18th, 2013 at 10:23 am PST) by Oscar

We are currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for the connection errors which customers on sub3/sub4 are experiencing at the moment. This is also causing webmail to timeout. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

Update June 18th 11:40am: We have received the “all clear” from all of our teams. All mail services are up and responding without any issues for homiemail-sub3 and sub4. If you are having any issues with your email now, please contact our support team as it is no longer related to this outage. You can contact our support from your web panel or click here. Thank you for your patience while we resolved this issue.

Update June 18th 11:20am: We are continuing to spot check all of the mail machines for sub3 and sub4. All mail services are up and responding now without any issues and once we get the “all clear” from the teams, we will completely resolve this post.

Update June 18th 11:00am: All mail services are up and responding now without any issues. Before we resolve this status post, we’re checking with all our teams to ensure there are no further issues. Once we get the “all clear” from the teams, we will completely resolve this post.

Update June 18th 10:45am: Our Data Center Operations team has isolated the issue to a popped circuit breaker upstream from our equipment. We are working with our facility to rectify this specific issue going forward. The affected machines are back online and our Systems Engineers are ensuring that all mail functions are working properly.

This entry was posted on Tuesday, June 18th, 2013 at 10:23 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

47 Responses to “Mail connectivity issues affecting homiemail-sub3 and homiemail-sub4 (RESOLVED)”

June 18th, 2013 at 10:26 amDreamhost Customer for Way Too Long Says:

Wow. 2 outages in less than a week. First my websites. Now my email. Go for the Gold Dreamhost. You win for sucking the most.

Would this also affect the ability to log in to panel? I can’t access my e-mail and I’m getting a login error with my account credentials at panel.dreamhost.com

What is the impact? lost mail? unable to receive? or just cannot log in….

Why do these mail issues keep happening?!?!?!??! I’ve been a customer since 2004 and have NEVER experienced this many problems with your mail. Get your s#&% together, Dreamhost!!!!

@Chip: This shouldn’t cause any problems with accessing the DreamHost panel, at this time it appears to be purely email related, and just those two email clusters.

YOU’RE THE WORST! Why does this keep happening?

Yeah this is getting out of hand and making my life hell. The stability (or lack thereof) of the homiemail-sub4 server has been unacceptable.

It’s not just mail. I can’t get into my panel or my website. It’s bad enough that my site is slow but not being able to reach it is just plain frustrating.

June 18th, 2013 at 10:33 amAngry Customer Says:

These are the same symptoms homiemail-sub4 has been exhibiting for weeks. You never completely fixed the issue posted at Jun 04
2:42 pm. It’s the same problem, going on week 3 now.

Can you please confirm the impact of this event, will emails sent during this outage will be lost, returned to sender or we just cannot access them.

We are getting “Unknown user or password incorrect” when logging in to any of our email accounts. Is this part of the problem? How long until rectified?

I do have to say this is getting annoying, but I do have to give you guys props for starting work on it so quickly. Within 3 minutes of me noticing it myself, this status update was already up. Anyway, hope this gets solved soon :)

Hmm, I don’t have an ‘Account Status’ tab in my panel? I can’t find anything about an email cluster anywhere either… I have a status tab, but it only has the following: Site Statistics, Data Centers, Bandwith Usage, Disk Usage, and MySQL Usage..

Been a customer since 2002. This may be my last month here. No one seems to care. Paying for top level service and I’m getting service I could almost get for free. HORRIBLE!

@Chris: it appears to be just a problem logging in

@JC: we’re still looking into this one, we’ll have more details soon but so far it doesn’t appear to be a repeat of the same causes we’ve seen previously

Been here since 2006. This is probably the most common thing I see. I must love to be punished as a customer since I have threatened to move, but have not and continue to pay 2x what I would pay at another host that I use for all of my customer sites. Those sites and email servers never have even half the issues I have seen here. Go figure.

Yes Dreamhost, please address the issue of WHY this keeps happening! It has been ongoing problems for homiemail-sub4 for WEEKS! This is unacceptable.
I agree with Angry Customer…

Here we go again.. Yet another problem….

Maybe fix the SSL errors that keep coming back?

Emails sent to my account were not received. They better not be lost!

@Amor. I was lucky enough to have a customer tell me that the email didnt go through and it tells them that the domain does not exist. So Im super excited now that my customers think Im a goner or some other random thought. I get that these things happen but it sucks even more knowing that the mail is just getting rejected

Well that is serious… I am waiting email for several thousand dollar projects.
Staff can confirm this, that emails sent during this period will be bounced or lost?

How long will this problem take to fix?!?!?!?!?!

June 18th, 2013 at 10:45 amZachariah Wiedeman Says:

I have clients calling me to ask when this problem will be resolved. Can you please update us with an ETA?

I know this isnt proof really and anybody could write it but this is what I got from my customer.

Hi Dwayne,

I tried to reply to your feedback message – but your email address is sending back all your emails citing that the domain doesn’t exist.

Deep 6 the homie cluster.
please move us to new hardware.

@Megan and Zachariah: our latest report from the datacenter says the power has been restored but a few of the mail servers need to be rebooted to regain access to those file servers, it will likely only be a few more minutes barring any more issues

Email problems again!? Come on DH! I’m patient and I understand that technical and hardware issues happens, but if they happen very often, or you have an extremely bad luck or you’re doing something wrong.

Mail is back but 1 hour of emails have been lost or bounced. This is indeed a serious event. Please use all means necessary for this do not repeat in the future.

OFCS. Again? Will this never end? How could two servers shutting off stop our mail? Isn’t there failover to other servers? This is egregiously bad reliability.

So it looks like mail is back up, but there is no email since about 8:30CDT. Am I to assume that anyone sending me email over the last 5 hours got an email that bounced back?

Bounces are better than the server eating it. But please advise what happens with email when a server goes offline.

Is there a way to recover lost emails from your side?

I’m sending a test mail to myself and it does not arrive.

Will the mail that was sent to my email addresses during the downtime be lost? Was it bounced, or is it still on the way?

I think my mail is slowly arriving. Disregard my last comment.

How can this happen? Dreamhost please give us some sort of assurance that measures will be taken to avoid anything like this in the future. For email to be down is one thing but for emails to actually be lost affects the business of so many people at so many different levels. It makes us look so bad to our clients that we depend on please throw us a bone here…

Still not resolved for me. 30+ mail accounts across 12 domains not working.

So the people that tried to email me during those times just got errors? Ugh! Bounced emails means lost clients for me. Thanks!

Tried sending myself some mail. It’s been dropped 7 times out of ten, without a bounce back or error message.

How are you still in business?

Homiemail-sub4 failing hard in Indiana… this is TURRIBLE…

Still not resolved for me either homie-sub4

I’ve been checking the mail logs and it appears most emails were simply deferred until the mailbox was accessible again. In most cases they would not have bounced.

June 18th, 2013 at 11:20 amLuis Serrano Says:

Issues with homiemail-sub4 again? what is wrong with you people?

Same here. homiemail-sub4. No bounce backs. Just lost emails. FML. I’m waiting for critical emails from two job opportunities and now I don’t know what to do. Email them that sorry my email host is incompetent and can’t get their crap together so I have to send this stupid email just in case you did reply to me? WTH!?

Are there any security concerns? I saw a dialog that said my mail was being accessed by another device during this outage.

I’m still having intermittent problems with one account

This problem is still happening. Can’t get access to my email. Please fix it asap.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking issues affecting both US-West Data Centers (Los Angeles and Irvine) (RESOLVED)
6:41 am

Networking issues affecting both US-West Data Centers (Los Angeles and Irvine) (RESOLVED)

Posted (June 13th, 2013 at 6:41 am PST) by Oscar

Emergency network maintenance: Due to the failed optical card on one of our core routers that affected our US  West Los Angeles and Irvine data centers earlier this morning, our network engineering team is going to perform an emergency maintenance to replace the failed component. This will help to solidify our network from this type of incident happening again.

We are estimating two hours for this maintenance, to adequately pre-test and make changes and test again. We don’t expect this procedure to cause any noticeable service interruptions. Once we have started, we will be monitoring your status post comments to make sure everything is going as planned. We will update this status post as soon as the maintenance commences and will be updating it with progress reports.

Update June 13th 10:15am PDT: The maintenance is now complete and we have received the “all clear” from all of our teams. All services are up and responding without any issues. If your sites are down or not resolving, please contact our support team as it is no longer related to this maintenance. You can contact our support from your web panel or click here. Thank you for your patience throughout the initial outage as well as the emergency maintenance. 

Update June 13th 9:45am PDT: The router component maintenance is now complete and we are spot checking all services to ensure that they are responding. Our preliminary checks are positive and once we receive the “all clear” from all of our teams we will completely resolve this status post.  Thank you for your patience while we continue with this emergency maintenance.

Update June 13th 9:40am PDT: The replacement components are in place and online. At this time we are rebalancing all traffic between the routers. This should help resolve the intermittent stability issues we have been noticing since the maintenance began. Thank you for your patience while we continue with this emergency maintenance.

Update June 13th 9:20am PDT: We are noticing some customer facing issues at this time and we apologize for the inconvenience. We are aware of the intermittent connectivity which is a direct result from this maintenance (we’re routing through our backup hardware while we prep the replacement). As soon as we’re able to cut over to the new hardware the current symptoms (due to the overload on the backup) should be alleviated. We will update this as we continue to progress with this maintenance.

Update June 13th 9:00am PDT: There has has been no impact as of yet with this maintenance. All services are responding, but we are rolling out this maintenance slowly to ensure we are covering all of our bases. We will continue to update this post as we progress.

Update June 13th 8:30am PDT: We are now starting the emergency maintenance. More updates will follow as we progress.

————–

We are currently experiencing network connectivity issues which is affecting two of our datacenters. This is directly affecting our Los Angeles and our Irvine data centers. Traffic between US-West and US-East are unaffected at this time. We are also seeing DNS issues on our authoritative name servers which may be the main cause. Our network engineering team is investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses. We apologize for the inconvenience this causes.

Update June 13th 8:00am PDT: The cause of this outage has been narrowed down to a failed optical card on one of our core routers. After restoring routing to the problematic hardware, service was restored. We understand that there is risk of this happening again, so our network engineering team is currently staging a replacement for that router. We anticipate that the other core routers will be able to handle the load during the actual time of replacement. Further details will be announced as we begin this network maintenance.

Update June 13th 7:40am PDT: We have narrowed down the intermittent connectivity issues to be hardware related.  Once we have more information we will update this post again with further details.

Update June 13th 7:20am PDT: We continue to investigate the root cause of this issue. As soon as we have more information we will update this post. Again, we sincerely apologize for the inconvenience this causes.

Update June 13th 7:00am PDT: Our network engineering team is still investigating the cause of this issue. At this time we have yet to find the root cause, but we will update this post as soon as we have more information.

This entry was posted on Thursday, June 13th, 2013 at 6:41 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

109 Responses to “Networking issues affecting both US-West Data Centers (Los Angeles and Irvine) (RESOLVED)”

Another day and another new problem! When will it end, oh when I find a new host… Google Search: reliable web host

I can’t access my ftp but im not going to cry about it … I been a dream host customer for almost 8 years now and I have not one bad thing to say about them.. I will be a dreamhost customer till I no longer access the internet, which means till I die lol love dreamhost keep up the awesome job guys :)

How is anyone supposed to host a website here? This just keeps on going down again and again.

Just Another Day !!!

I’m auditioning for a new host. Soon I will make the migration.

Again with this? Is there just something direly wrong with your hardware? This is making myself and my clients pretty nervous.

Can’t access FTP, not going to even try making changes on sites until there’s a resolution.

It’s okay. Don’t use Dreamhost if 99% uptime is not enough. Good luck fixing it guys.

We do apologize for the problems. For those of you with issue accessing FTP, can you let us know the exact error you’re getting when you try to connect? Are you able to visit your sites at all?

This is getting ridiculous. Once every couple weeks, my site is down.

June 13th, 2013 at 6:54 amDan Wigglesworth Says:

Hang in there! We need you!

i was already loading slow. now not at all. boo

June 13th, 2013 at 6:55 amDan Wigglesworth Says:

Is there a twitter status feed? That would be helpful.

June 13th, 2013 at 6:56 amDan Wigglesworth Says:

Don’t get hung up on the root causes just yet… getting it to barely function for the moment woudl be quite enough for me! Root cause analysis can wait! :)

@ Brian@dreamhost

im getting can’t connect to server timed out when I try to access my ftp hope this helps :)

Requested a password change for user(FTP) account and its pending for at least 30 minutes.

When will I learn. These constant outages and interruptions are killing my business. Must. Leave. Dreamhost. Now!

@Dan Wifflesworth: we update https://twitter.com/dhstatus. I’m working on having it updated to reflect this issue now. We don’t do play by play like we do on this post, however.

@Michaelg, it does, but I need to know if you can access your site as well!

this really isnt a big issue, my site has been slow for a very long time but i didnt expect “super fast” with shared hosting, and really if i needed faster id upgrade to a vps.

as for the current issue, man some of these comments make it out to be a daily thing, im thinking there just shills for competitors. because in the 6+ years ive had hosting with DH ive rarely had issues, maybe once a year at that.

as for folks going on about 99% uptime, you really might wonna do the math on that it comes out to 3.65 days of down time per year, or 7.20 hours per month, or 1.68 hours per week. based on that id say i at least have gotten better than 99%

FTP getting connection time outs and failed to retrieve directory listing. Can generally ftp to my root just fine. But after that, usually times out trying to enter any directory

Most of my websites are actually loading faster now compared to the last couple days. I must be the lucky one.

@ Brian@dreamhost

yes I can access and see my sites I just can’t connect to my ftp it asks for my username and password then gives me a time out can’t connect to server ..

My sites were down for about ten minutes but back up now.

BTW, been here for almost ten years. Go find your more reliable host for the money you spend. Good luck with that.

All my sites on professorx are down! Any progreess, please?

All my sites are down. You can barely browse two pages before the connection tanks. Any idea when this is going to get fixed.

June 13th, 2013 at 7:11 amDan Wigglesworth Says:

@ Brian@dreamhost

Thanks for the twitter feed. I just RSS’d this thing … but the refresh is set to 1 hour? hmmm…

DIAGNOSTIC INFORMATION: I can access my site on the shared service, but Twilio (telephony services) isn’t getting a fast enough response (not that my twilio stuff is very demanding — just a couple of phones going on)…. So twilio is throwing errors instead of routing my phones (yes — I have a backup service configured).

NOTE: I had a similar problem just a few days ago and one of your staff spoke to me on the phone and very helpfully blocked a bunch of Chinese IP addresses that were busily querying my site. That definitely fixed the problem back then. I presume this is NOT related… but … well… you do the math.

@Brain@dreamhost i’m the same as Michaelg site is slow but alive.. ftp and mysql access is dead and times out!

When trying to login to my site I get 2 different issues.

1) Error trying to connect to database
2) Typical ‘could not connect to website’

My FTP appears to be working fine, but I didn’t try to actually upload anything.

I have been slightly annoyed with the multiple issues lately, but our company has used dreamhost for 6 years and during all that time we maybe had 2-3 small issues (like 2-4 hours each). However the last month or so we have had a lot more than usual. I think we are spoiled on 99.999% uptime and are getting annoyed at a mere 99.5%.

Error establishing a database connection here
I hope you guys fix it soon

MikeFreck Says:

@Brain@dreamhost i’m the same as Michaelg site is slow but alive.. ftp and mysql access is dead and times out!
**************************************************

yes that’s my problem I can’t connect to MySQL either, times out like my ftp .. also my site just went down can’t connect to it either now..

I’ve been a Dreamhost customer for 9 years and have never seen as much slowness and downtime as has happened over the last couple of months. When I complained about all the current email issues and explained that dependable email is critical to my business, I was told, “Dreamhost is in business as a web hosting service, not email.” That type of response is completely unacceptable, as are the constant slowness and downtimes. I will be changing hosts as soon as possible.

Sites are working from me same for the db but ssh and svn server is down on port-au-prince. Any updates please?

My VPS has at most intermittent t connection to Akismet. Mostly what I get is:

Unable to find Akismet servers.

A DNS problem or firewall is preventing all access from your web server to Akismet.com. Akismet cannot work correctly until this is fixed. Please contact your web host or firewall administrator and give them this information about Akismet and firewalls. [link to http://blog.akismet.com/akismet-hosting-faq/

@Marion, the response you received about your email isn’t one we endorse, all services we offer are important to us. Can you provide your domain name so I can look into that for you?

Also, the mysql login problems you guys are seeing are known, they’re due to the connectivity between datacenters. We’re working on that, and apologize for the problems!

Sites now working professorx- thx

I would not mind but these things ALWAYS seem to happen when I need to do something important, I have a wordpress site that is down because it can’t see the database, do I blame this fault or should I assume my site has been hacked? How come some of my sites work and others don’t. Perhaps this error has nothing to do with it.

Error establishing a database connection

This either means that the username and password information in your wp-config.php file is incorrect or we can’t contact the database server at http://bit.ly/hostga. This could mean your host’s database server is down.

Are you sure you have the correct username and password?
Are you sure that you have typed the correct hostname?
Are you sure that the database server is running?

Overall I rate Dreamhost and will forgive them if one site is affected for a short time but if this continues it is going to get me angry, last time I went and got a backup account elsewhere, but I hate cPanel, the Dreamhost panel is SO much better.

all my sites/databases on conklin have been down for almost two hours now. ftp and email are working but sites and databases are all down.

June 13th, 2013 at 7:42 amDan Wigglesworth Says:

Seems to be back to normal now. Twilio is getting a timely response from the (shared server) website.

Thank you!

yes everthing seems to be back to normal on my end also :)

thank you :)

great work, seams to work

These are the errors I’m seeing on a couple of my sites. Hopefully it helps diagnose the issue.

1. SQL ERROR [ mysql4 ]

Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110 [2013]

2. Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@nebraskagbody.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

fiuuu!!, you save my job, thank u’

>Tiziano Says:
>It’s okay. Don’t use Dreamhost if 99% uptime is not enough. Good luck fixing it guys.

99%? lol.

Regrettably, I signed up for a hosting package yesterday for a domain I needed to publish quickly. Although my payment has been posted my order is still in ‘pending’ mode. I’ve tried getting in touch with someone at Dreamhost, both by placing a trouble ticket and requesting a phone call, but have not received a response, so I tried to follow up earlier this morning only to find their website was offline due to a network problem. This is incredible!

All I want at this point is to cancel the order and get a refund. I used the ‘cancel’ feature on the control panel (once their website came back online) but keep getting an error message that the server can’t be reached.

And this company stays in business how?

Just to keep everybody in the loop, we’re still in the process of staging a replacement router and will be performing emergency maintenance shortly to replace the one that failed. We’ll be posting updates here and will make an announcement as soon as we have more information on when that will happen.

@Susan – I’m sorry for your rocky start. If you could let me know the domain you signed up for I’ll be happy to make sure someone addresses your support ticket.

In case you missed the update above, we’re starting the maintenance now. We don’t expect any downtime as a result, but will be monitoring comments here in case anybody is experiencing any issues.

My site is still down.

June 13th, 2013 at 8:39 amIn Disbelief Says:

Setup space on another hosting company with GREAT service and NO outages and no DAILY issues on a posting message board. And it’s 9.95 per month! BYE BYE DREAMHOST !!!

I am back up and running! Thanks for the quick fix dreamhost!

@Gina – your site outage is unrelated to this issue. We contacted you two days ago with information regarding your account. Please log in to your web panel (panel.dreamhost.com) and go to the ‘support history’ section to see the situation with your account.

Seriouslly? again? put your shit together dreamhost

web:
Service Temporarily Unavailable

phpMyAdmin:
#2003 – The server is not responding

Are we seeing the same issue again? It seems all the problems I had this morning went away and then came back about 5 minutes ago?

Seems Braves shared sever is down!
My website doesn’t seem to be working at all.

Online business is really hurting right now !!!

Checking e mail using POP isn’t working or sending it. Please don’t make us go through this all over again

i have noticed an interruption.

My site and FTP are both down right now. FTP error is “Can’t connect to `173.236.174.211′: Operation timed out. Web seems to be a timeout as well.

all sites are not responding at this time

Cannot connect to our site. No errors, it just never connects and times out. Cannot connect to the server.

again and again..

Another longtime customer here. I’ve had many years of trouble-free service from Dreamhost, but the last few weeks have given more issues than I’ve had in the years before.

That said, anyone who thinks that the grass is greener elsewhere really hasn’t tested the market. I’ve been burned by more dodgy operations than I can remember, but Dreamhost has been a quality operation for as long as I can remember. It’s gonna more than a couple of hiccups to make me switch.

We’re down – AGAIN. This is not good, we host our helpdesk here and I can’t guarantee to my clients that their tickets will be received and dealt with. This cannot go on. Like many others this potentially can cost us clients.

My site is down right now. I’m in Irvine so I’m not sure if that has anything to do with not being able to access it.

We’re running into some issues with the maintenance and it resulted in some unexpected downtime. We’re working on it and will be updating the post as we have more info available to us. We’re sorry for the inconvenience this causes everybody.

Mail is down here…. just sayin

My sites’ http, FTP and webmail are all down.

Sites not coming up (this and all subdomains). FTP connection times out.

Mail not working as of about 15 minutes ago. Receive error that mail server cannot be found for sending and receiving.

Hi, my sites fine but only one (and most importand for me) down and give the database error message. Please help and fix…

Site off line here.

Mail just went down. POP/IMAP/SMTP.

This sucks big time! All my sites are down.

Can’t access either of my sites. One is VPS (which can’t connect to the MySQL server), the other is on Galaxy.

No interruptions? All sites and FTP are down.

30 mins later…

Oh my god – You lied, my internets is down.

“In case you missed the update above, we’re starting the maintenance now. We don’t expect any downtime as a result, but will be monitoring comments here in case anybody is experiencing any issues.” – danwy

site is also down. thanks a lot.how does your equipment fail every week. it really has been every week all year there have been issues

As I mentioned in a post above, we ran into some unexpected issues with the maintenance that has resulted in some downtime. We’re aware of the issue and are working to get it addressed ASAP. Stay tuned for more updates.

If you really need a host that’s up a majority of the time, check out FlipHost at fliphost.net. I moved a bunch of sites from DH to FlipHost about a month or two ago and have never been happier. Their VPS prices are unbelievable. I’ve had a total of about 3 minutes of downtime since switching to FlipHost.

I still have some sites at Dreamhost though, they’re not super important and can deal with frequent downtime. Dreamhost offers a good service if you don’t run sites that need to be highly available. If you’re running a business off the shared hosting platform, probably time to look at other options, though.

Our site is down too. Blerg.

Folks,

I’m almost a new comer to Dream Host and I’m getting nervous due to the repetitive downtime. Please let me know how often do you face such problems? How do yor deal with youe clients/users?

My site is down; timeout error; it’s an ecommerce site and Thursday is usually an awesome sales day. Whatever you can do to fix this quickly is appreciated. Thank you! :)

Yup – seeing it here too. Intermittent connectivity issues.

Thanks.

“We are noticing some customer facing issues at this time and we apologize for the inconvenience” Facing issues is putting it mildly considering what everyone has been going through the past few months. I think I speak for most people when I say stop apologizing and just make things stable.

This is why this is my last year with DreamHost (after putting up with them for many years). A blip here and there is okay, but this is plain ridiculous!

How do you deal with your clients/users? You explain that nothing is 100% uptime. That’s how the internet works. Deal with it. If they keep bitching, you could try hosting somewhere else, but guess what? They won’t be up 100% either.

This is a huge LOSS to online businesses.

Unacceptable hosting the dream dreamhost unacceptable.
I think we all demand credits for the loss we are incurring here.

We needs the monies, we needs the monies, we need the monies!

Oh, and I’ve been with Dreamhost since 2002.

Well, it is the end for me. i will migrate all my important clients to amazon web services.
Every time dreamhost go down my clients lose money so they will be happy paying a little bit more for a better service that keep the sales going up.

Good Bye

We needs the monies! We needs the monies!

Hole shit man, my web sites are down!!!! $$$$$$$$

ps9411 unreachable!

Now everything is down on ps41356.

There are SOOOO MANY cheap VPSes out there but you guys just keep sticking with DreamHost as if it’s the one and only host you ever heard of in your life….. :))
What do you say about 2GB RAM dedicated VPS, 2TB Monthly Bandwidth and more, all for only $7 month???

Check this out: http://www.lowendbox.com/

Its a site that reviews and offers promos to many VPS companies!

DREAMHOST heard me & will be crediting a FULL year of service!

Demand it! – we needs the monies!

Email is now working.

I guess I need to spring for a VPS (and not on Dreamhost) I’m cost-conscious (obviously or I wouldn’t be here.) Does anyone have any good advice? I’m considering MediaTemple.

My email (IMAP and webmail) and website are down, this is really bad for my business.

Websites have been restored too!

YAAAAY!!!! its working!

claps…..but just once for hosting the dream – dreamhost. keep it up pleeeez.

We just passed the halfway point of the maintenance and are in the process of balancing traffic between the backup and new router. Things are starting to look better and should only improve as more and more gets transferred over. We’ll keep everybody posted as this continues to progress.

The Sr. Mgmt / Leadership needs to be replaced. This company is going down hill and new team needs to come in and bring it back to its old glory.

I see my FTP is back up on monrovia. My question is why doesn’t spare equipment kick in and auto-failover? Then the admin should get an e-mail to know to go replace the downed equipment. I’m sure they do something like that for bad hard drives. But there should be all kinds of redundancy for hosting millions of websites. I’ll say I am comfortable with sticking with DreamHost since I get that they just have a massive scale of customers to support.

I have been here for 11 years now and the last year has been the worst by far..
I have no other host, and we use 4 others depending on importance, that have ANY hardware failures I even am aware of… this is the end.

My website seems to be working now, it was down a few minutes ago. Some of the messages I’ve sent during the downtime through my jabber account have been silently lost (they have not reached the destination and I have not received an error).

@The Jedi – our setup is redundant so backup does kick in automatically. In this case a backup that we use to help share the load had the failure. We had to move hardware from our lab to fix this. Please keep in mind that we have also not completed our Network Maintenance Series – which is a total overhaul of our network. We are still on a temporary solution until that maintenance series is complete.

@R unfortunately jabber service is known to lose things in the void if the service is having connection issues. You should be back up and working fine now. Sorry about that.

@Ralph

“Please keep in mind that we have also not completed our Network Maintenance Series – which is a total overhaul of our network”

Hey Ralph – Will this be done in parallel to the already running temporary solution & port over when the time is right or will you guys shut everything down then work on the more permanent solution?

If it’s the latter then I assume the downtime for that “overhaul” will be significant moving forward.

thoughts?

@michael – the plan is to stage the setup completely and then move each vlan piece by piece. We have been able to do that successfully in previous maintenances with the exception of undocumented limitations we bumped in to in some of our most recent Network Maintenance Series updates. With the knowledge gained, we feel good about that we can do it without an “all dark” maintenance. We understand and have lived through these network changes and I’d like to say we will have the least amount of service downtime possible when we move forward with the permanent solution. I will do my best to make sure, once we revisit the network maintenance, that we detail as much of our plan as possible.

As always, THANKS for fixing the problem with your usual speed, Dreamhost! :)

Client email is not coming through to me and is getting bounced back.

WTF?

MY mobile mail has not worked right since you upgraded (???!) the server.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Package Upgrade on Dedicated Servers – Friday June 14th 11am PDT (RESOLVED)
3:13 pm

Package Upgrade on Dedicated Servers – Friday June 14th 11am PDT (RESOLVED)

Posted (June 12th, 2013 at 3:13 pm PST) by Oscar

This Friday we will upgrading the fcgi package on all dedicated servers. We will be starting at 11am PDT on June 14th   and anticipate this upgrade completing on June 15th. We do not anticipate any downtime for any of our dedicated customers during this upgrade.  Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

Update June 14th 2:15pm PDT: The package upgrade was completed earlier this morning and after final checks, we are now closing this status post. The upgrade was completed without any issues, but required apache instances to be rebooted. If you have any issues with your dedicated servers or sites, please contact our support from your web panel or click here.

This entry was posted on Wednesday, June 12th, 2013 at 3:13 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West (Irvine) Wednesday June 12th 10pm PDT (3 hours)(RESOLVED)
5:31 pm

Network Maintenance US-West (Irvine) Wednesday June 12th 10pm PDT (3 hours)(RESOLVED)

Posted (June 10th, 2013 at 5:31 pm PST) by Ralph

This Wednesday we will be performing maintenance on the network at our US-West (Irvine) data center. We will be starting at 10pm PDT on June 12th with a 3 hour window. During the maintenance we will be modifying the links between our border and core routers, as well as our link connecting our data centers. The redundant network should give us the ability to perform these changes without impacting your service. Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

To see if you’re affected by this maintenance, please check your location in the DreamHost panel.

Update June 13th 12:15am PDT: Okay we are resolving this maintenance. We have successfully completed the maintenance. Thank you for your patience throughout this update!

Update June 12th 11:25pm PDT: The panel issue we experienced is now resolved. It was not due to this maintenance, which is still underway. We have corrected the database issue that caused any panel issues noticed and we will be watching that closely.

Update June 12th 11:17pm PDT: We are currently experiencing issues with our panel. We are working on it right now and hope to have it resolved soon.

Update June 12th 10:45pm PDT: We are nearly finished with phase 3. We have some final configuration changes to make. So far we have not seen any impact from this maintenance.

Update June 12th 10:20pm PDT: Phase 2 has just completed. We are about to start phase 3. So far we have been able to perform the maintenance with 0 impact to services. We expect phase 3 to go as smoothly as phase 1 and 2.

Update June 12th 10:10pm PDT: Phase 1 has completed without incident, as expected. We are moving on to phase 2 of this maintenance now.

Update June 12th 10:00pm PDT: We are beginning phase 1 right now. As stated our plan is to use the redundancy in our network to perform this maintenance with 0 impact of service. We will update this status post as we progress through the 3 phases.

Update June 12th 9:50pm PDT: Our team is ready for maintenance action. This will be a 3 stage process. We will be starting phase 1 in roughly 10 minutes. We do not expect any service interruption. Comments are now on and we will be evaluating them.

This entry was posted on Monday, June 10th, 2013 at 5:31 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Network Maintenance US-West (Irvine) Wednesday June 12th 10pm PDT (3 hours)(RESOLVED)”

It seems the control panel is down… At least intermittently but I haven’t been able to access it tonight (https://panel.dreamhost.com)

@Jeremy – we are looking into this now. We are working on it. We are unsure if it’s related to this maintenance just yet.

Oh OK. Well at least there’s a message there now (instead of timing out). ;)

… And @Ralph – thanks for the response, btw.

A few minutes ago I got “Can’t call method “prepare” on an undefined value at /usr/local/lib/site_perl/Ndn/Common/Object.pm line 294.” when trying to switch to panel.dreamhost.com/index.cgi?tree=users.users& I see it’s being worked on though.

Yep, the Panel is toast. It just timed out. Will see if that subdomain I just made really got made.

11:27 pm PDT. Panel up!

@everyone – panel is back up, as unrelated and due to a database issue, which we have cleared up. Sorry about the scare there.

@Ralph – Thanks!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Outgoing mail delays, slowness, and disconnection errors in ‘homiemail-master’ and ‘homiemail-sub4′
2:42 pm

Outgoing mail delays, slowness, and disconnection errors in ‘homiemail-master’ and ‘homiemail-sub4′

Posted (June 4th, 2013 at 2:42 pm PST) by Seohee

Our system engineers are noticing elevated loads on some machines, which is causing a delay in the processing of sent mail as well as slowness or intermittent disconnections in your email client and webmail. Our administrators are working diligently on getting the loads down, as well as working on a longterm solution to improve the stability and service as a whole. Unfortunately, there can be intermittent reoccurrences until this is complete.

You can log into our Web Panel and confirm whether you are impacted by this outage. Click the “Account Status” link. Again, you are only impacted if “homiemial-master” or “homiemail-sub4″ is listed in the field labeled “Your E-mail Cluster”.

We will provide you with progress updates on the status of the longterm solutions on this post. We sincerely appreciate your patience and are sorry for the trouble this has caused.

Update June 15th, 4:56 PM PDT: The new setup and the new machines are making serious improvements on the delays. We are still considering solutions for the longer-term, but for now you should not be seeing issues — if you are, please let us know here. Thank you for your patience!

Update June 7th, 3:27 PM PDT: We have made some changes within the setup of the mail servers, and we have added a few machines to help with the mail delays. We are still working to completely resolve the issue, but we are making progress! Since this is an ongoing project, we’ll keep you informed!

Update June 5th, 4:52pm PDT: We’ve found one of the causes of the intermittent slowness on your email cluster, and made the necessary changes to speed things up again. We’re continuing to troubleshoot and find ways to improve our mail services, and appreciate your continued patience. -Seohee

Update June 5th, 2:10pm PDT: We’re very sorry for the lack of updates regarding this issue. The high server loads mentioned in the post above are what is causing the connection issues and mail delays you’ve been noticing. Our admins have continued to work on these servers over the last 24 hours and have made quite a few changes during this time. We found an update made to our load balancer that wasn’t working as intended and causing mail to not be properly routed to the least busy server. This ended up overloading some of the mail servers and drove up the load on them. In addition to that change, we’ve also deployed several more servers to this cluster to help out with better distributing the mail load on the cluster. There are still a small amount of servers that have a high load on them, but we are addressing those individually now and stability is slowly improving.

These are all just steps to address the current issues though. In addition to these steps that have already been implemented, our admin team has been hard at work planning a complete restructuring of the mail cluster to improve its stability going forward.  We are still in the early phases of planning, but we are looking at past issues we’ve had with mail so that we can avoid many of the things that are causing issues on the current cluster.

Again, we apologize for the inconvenience this has caused everybody. We understand that Email is a very important service and we are doing everything we can to make sure it remains stable.

This entry was posted on Tuesday, June 4th, 2013 at 2:42 pm and is filed under Announcement, Email Issue, General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

44 Responses to “Outgoing mail delays, slowness, and disconnection errors in ‘homiemail-master’ and ‘homiemail-sub4′”

This is not a new issue. I first reported this yesterday and the problems still continue.

This is the third or fourth email issue this year. Why are the problems still occurring? What measures are you taking to make sure the problems are permanently resolved?

More importantly: has inbound email been bouncing back to the senders? Or is it just getting dropped again?

I can get to my website, but I cannot get to my dashboard. I am also unable to upload files via filezilla; and unable to connect to server. Are you down?

Yesterday I had email from a client bounce, so they sent it again and it got through. I am on homiemail-master. Today email to a different client from one of their staff bounced. They are on homiemail-sub4. I don’t remember this ever happening since I started with DreamHost in 2007, but it appears that DreamHost’s infrastructure has been crumbling lately. Are they going to be more proactive and not so reactive, or is it time to look for a host with more reliable infrastructure?

Everything was going great until I had to drop a heavy deuce. It took me about 30-minutes to work up enough anal mucus so I could deliver that son-of-a-bitch proper. When I was able to finally clean myself up and wipe the sweat from my brow, I return to my computer to find that everything stopped working. Help?

The 2 bounced emails were both sent from Hotmail, and DreamHost support says:

This is a problem I’ve seen before with Hotmail. Your domain has two DNS records, one for the website and one for the mail, and Hotmail sometimes makes a mistake and sends mail not to the mail server but to the website server. This was not in any way caused by the recent mail server issues here at DreamHost, this was an error made by the Hotmail server when deciding where to route the email.

Wow! Sorry for blaming DreamHost when our pal Micro$oft wrecks things for us all again.

Hi, right now homiemail-sub5 is down!!!

This has been going on for weeks in varying degrees. SORT IT OUT.

What’s going on here, this is not an isolated incident, issues with mail have been going on for weeks. What is the update?

My Website AND DH Panel is down… Lovin’ it!

Website down (errors 500, 404,408). :(

Still down, no email, no website, no webmail…when will this get fixed?

My email cluster is homiemail-sub5. I am receiving email, but the senders are getting this error message anyway, which is causing me problems:

This is the mail system at host homiemail-mx20.g.dreamhost.com.
I’m sorry to have to inform you that your message could not
be delivered to one or more recipients.
homiemail-mx20.g.dreamhost.com

This is one of MANY email problems we’ve had with DreamHost this year. Incredibly frustrated.

You have to be kidding me. AGAIN!!!!

We’re having the same issues on homiemail-sub4 here in central Indiana. Agree with Edmuch and Zach above– we have been having increasing issues the past few weeks. This doesn’t seem to be an isolated incident!!

How about an update?

We can’t send our mail merge.

homiemail-sub5 down for me here too.

It is ironic that Dreamhost asks for our e-mail address before posting here, given the % of downtime we all seem to experience with the service. Homiemail-sub5 seems to have more than its share of issues. I seem to recall it was replaced or upgraded or something but that doesn’t seem to have done the trick.

I’m in EDT. For what it is worth, as of Monday June 3, my mail has either crashed or slowed to COMPLETELY unusable right after lunch. In other words, when the other half of the continent signs on. I rely on e-mail as a lifeline with my employer and my contacts. This MUST become more stable and consistent. Completely unacceptable.

Our autoresponder is not functioning.

PLEASE give us an update on this. I am on homiemail-sub4 and am unable to communicate with clients. Yesterday was an all-day battle to send time-critical emails, and I no longer trust that any of them got through. Major disappointment.

I have another domain on homiemail-sub5 – it is also now experiencing issues.

sending mail using stmp.fpes.com fails using mac mail. we are on homiemail-sub4

It is getting worse here– the squirrelmail web client is now failling hardcore. I echo the above– we’re a small business and we rely on email in a major way! Help!

I think I’m able to receive mail. Outgoing messages are disappearing. As in, they are being sent out but not getting delivered. Any updates

Please post a status update.

It’s been almost 24 hours since this issue was posted here. The issue seems to be getting worse at the moment.

This really should not be going on this long. Can you please update status?

this is not what I paid for.

HAVE to have an update on this. Are clients getting emails? Will they eventually? Test emails to my other accounts aren’t showing up. Ugh.

Does anyone from Dreamhost read these comments?
I’ll bet they would if they got posted on the dreamhost home page…

The quality of Dream Host looks to be getting worse and worse. Yet again, there are problems.

I wonder if the CEO realizes it is the people and equipment that make up hosting companies. This means you need to invest in them and replace and upgrade them from time to time.

These mail issues have been near constant in the past month, and nearly every time we’ve reported them and expressed our dissatisfaction, we’ve been told that Dreamhost’s focus is on Web Hosting and not Email Hosting, and we should consider hosting our email with a 3rd party.

Well, then maybe you shouldn’t include email service with your hosting plans anymore if that truly is the case. If you offer web hosting and include email hosting as well, then YOU SHOULD PROBABLY TAKE ONE SERVICE AS SERIOUSLY AS THE OTHER.

Again with problems. I’m looking for another host.

Is anyone interested in filing a class action lawsuit???

We’re sorry for the lack of updates, but we don’t have much new to report at this time. We’ve been investigating the server loads that are causing these issues and have a few fixes that we’re ready to start deploying. We’re working on getting together enough information to update to this post and as soon as we have any relevant information we will be sure to pass it on. If you’ve contacting support about this issue, please make sure to provide as much information as possible so that they can continue to work with our admin team to resolve these issues. We’re very sorry for the inconvenience this is causing everybody and we are doing all that we can to get it under control ASAP. Stay posted for more info.

Haha, good fucking luck with that John. Read the TOS.

could we get a status update, please? my outgoing email’s still not working, not since early yesterday afternoon. (homiemail-sub4) thanks.

I am on homiemail-sub4 and have been getting a lot of “server connection failed” using the webmail app. The website is holding fine right now, but the email is really slow (delayed). Just wanted to put my two-bits in. Hope this gets fixed soon.

inbound emails are being sent back to senders

We’re very sorry for the lack of updates regarding this issue. The high server loads mentioned in the post above are what is causing the connection issues and mail delays you’ve been noticing. Our admins have continued to work on these servers over the last 24 hours and have made quite a few changes during this time. We found an update made to our load balancer that wasn’t working as intended and causing mail to not be properly routed to the least busy server. This ended up overloading some of the mail servers and drove up the load on them. In addition to that change, we’ve also deployed several more servers to this cluster to help out with better distributing the mail load on the cluster. There are still a small amount of servers that have a high load on them, but we are addressing those individually now and stability is slowly improving.

These are all just steps to address the current issues though. In addition to these steps that have already been implemented, our admin team has been hard at work planning a complete restructuring of the mail cluster to improve its stability going forward. We are still in the early phases of planning, but we are looking at past issues we’ve had with mail so that we can avoid many of the things that are causing issues on the current cluster.

Again, we apologize for the inconvenience this has caused everybody. We understand that Email is a very important service and we are doing everything we can to make sure it remains stable.

(Mostly) unrelated, but does anyone know if there are East coast mail servers? I couldn’t find that info in the obvious spots…

I do not think they have east coast mail servers. ALL of their mail servers are in CA (i believe). I agree they should most likely have two spots.

Thank you for the update!

We’ve found one of the causes of the intermittent slowness on your email cluster, and made the necessary changes to speed things up again. We’re continuing to troubleshoot and find ways to improve our mail services, and appreciate your continued patience. -Seohee

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects service disruption (RESOLVED)
3:50 am

DreamObjects service disruption (RESOLVED)

Posted (May 30th, 2013 at 3:50 am PST) by chalise

We are currently experiencing problems with our DreamObjects service. Our team is looking into this, and we will provide further updates as more information becomes available.

Thanks for your patience!

Update May 30 4:45am PDT: All services are now back online, and our team will continue to look into the cause, and how we can prevent any similar issues in the future. Thanks for your patience while we got everything back up and running!

Update May 30 4:30am PDT: Most services are back up at this point, but we are continuing to monitor it and make sure everything is and will remain stable. More updates to come.

This entry was posted on Thursday, May 30th, 2013 at 3:50 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Alchemy (US-West Irvine Data Center) UPS power and battery preventive maintenance Tuesday, May 28 11pm – May 29th 7am Pacific Time (RESOLVED)
6:42 pm

Alchemy (US-West Irvine Data Center) UPS power and battery preventive maintenance Tuesday, May 28 11pm – May 29th 7am Pacific Time (RESOLVED)

Posted (May 23rd, 2013 at 6:42 pm PST) by Ralph

Our data center provider, Alchemy, will be conducting preventative maintenance on their UPS power and batteries in the US-West (Irvine) Data Center. This maintenance will require the building UPS system to be transferred from the city to the generators during part of the window. Once complete the power will be transferred back to the city power again.

A UPS system is the battery backup that sustains the load of the data center when utility power is cut. Simply put, the job of the UPS system is to keep the lights on when transferring from utility power to generator power, and back again. Think of it as a battery backup for your computer on an extremely large scale.

Alchemy states:
“The maintenance should be transparent to you and no interruptions of service are anticipated; however as with any maintenance, there is a small possibility of service interruptions as changes are made. We expect the duration of the work to be approximately eight (8) hours.”

We will have our teams on hand to make sure, that should any interruption occur, we will be ready with a plan to immediately restore service. At this time we are confident that there will be no service interruption, however we plan to be prepared and have our teams responding to comments to this post and updating you with anything we hear during the maintenance.

To see if you have services in the US-West Irvine data center, please check your data center location in the DreamHost panel.

Update May 29th 2:42am PDT: Maintenance has now completed, without any service disruptions!

Update May 29th 1:42am PDT: Maintenance is now 75% finished, and everything is continuing to move along smoothly. More updates to come as the maintenance continues.

Update May 29th 1:00am PDT: So far, so good! We’re continuing to monitor this and have not yet experienced any issues affecting our servers or connectivity. We’ll continue keeping an eye out, and updating you every so often, though, just in case! It looks like we’re about halfway done, at this point!

Update May 28th 10:55 PM PDT:  We’re here, and on alert, in case our data center provider, Alchemy, needs us. We’ll keep this updated as things happen.

This entry was posted on Thursday, May 23rd, 2013 at 6:42 pm and is filed under Announcement, Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail connectivity issues affecting homiemail-sub4 and homiemail-sub5 (RESOLVED)
12:22 pm

Mail connectivity issues affecting homiemail-sub4 and homiemail-sub5 (RESOLVED)

Posted (May 22nd, 2013 at 12:22 pm PST) by Oscar

We are currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for the connection errors which customers on sub4/sub5 are experiencing at the moment. This is also causing webmail to load slowly. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

Update May 22nd 1:50 PM PDT: We are no longer seeing any intermittent connectivity issues. At this time you should be able to connect to your email clients and webmail without any issues. Our network engineering team is monitoring all mail machines to ensure this issue doesn’t pop up again and is investigating options to improve overall mail stability. We apologize for the disruption of service as we understand how important mail is.

Update May 22nd 1:25 PM PDT: We are no longer seeing any intermittent connectivity issues. At this time you should be able to connect to your email clients and webmail without any issues. Our network engineering team is monitoring all mail machines to ensure this issue doesn’t pop up again. We will continue to look into tntil everything is fully resolved.

Update May 22nd 1:05 PM PDT: Our network engineering team continues to investigate the cause of this issue. These mail connectivity issues are causing timeouts in both webmail and email clients. We will continue to update this post as we receive more information. We appreciate your patience!

Update May 22nd 12:45 PM PDT: Our network engineering team is working in the process of tracking down the cause of this issue. At this time we are noticing several mail machines with extremely high loads. We will continue to update this post as we receive more information. Thank you for your patience!

This entry was posted on Wednesday, May 22nd, 2013 at 12:22 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

24 Responses to “Mail connectivity issues affecting homiemail-sub4 and homiemail-sub5 (RESOLVED)”

this has been going on for 6 hours today on and off and was half of yesterday as well. can’t believe its just being posted.

And we go to another rain calls asking why the email is slow to arrive.

Sites are still slow when accessing DBs and the email is shot!

@brian: Our techs had been seeing some higher than normal cpu load on several mail servers but we were able to clear that up yesterday. It seems to have returned now which is why we’re put up this status post. We have several members of our network and system administrator team looking into the cause at the moment

@Em: This status post is only for mail related issues. If you’re also seeing slowness between your website and database servers then please contact support so we can look into that for you.

Thanks Dan. This is just frustrating as we have lost 2 partial work days this week due to dreamhost issues and over the past few months, probably 5-7 days.

I have been having issues with email for the past 2 days. I have put in 2 seperate support tickets and was told it was being addressed. Email is still not working. Really frustrating.

Can’t you spend money on more reliable equipment? I hate to say it – I’ve been a customer of yours since 1998 – I’m about to jump ship. I NEED reliability and my services to work all the time.

“higher then normal?”

Isn’t it obvious that the load you’re seeing today IS NORMAL!?!

So instead of reacting when the load gets to where it is, provide the bandwidth to handle this amount and then you can observe when the load is LOWER then normal and your CUSTOMERS AREN’T MISERABLE!

@John: We believed those issues were behind us, but since they have popped up again we are now looking deeper into rooting out the cause and hopefully getting this resolved entirely

@Justin: We’re not sure at this time if a hardware issue is to blame here, though it’s definitely something we’re looking into.

@david: That’s why it’s so confusing, the traffic to the mail servers shouldn’t be much higher than it was last week yet it now appears to be causing the mail servers to have high cpu load. Our network team are trying to diagnose if there is a malfunction or some other change that may have caused this.

I have been having the problem off and on for a couple weeks now on homiemail-sub5.

even with extra traffic i’m confused how nothing is coming through at all. should things be coming through slowly at least? no emails now for over 2 hours

saw here Amanda but its been a few months for us, not weeks

It has been awhile, seems to be getting worse. It is a real pain to try and get work done.

Why does it ALWAYS seem to be hitting Homiemail-sub 4? And WHY do I have to be on it? Maybe there is a bigger problem… I have been experiencing major delays in my email off and on for over two weeks! After not having any problems for the longest time, this is becoming unacceptable!

I have an email on homiemail-sub4 that works that was set up over a year ago… i just recently set one up on sunday and it hasnt worked yet…

i hope this info helps the debugging process!!!!!!!!!!

said service was mostly restored 30 mins ago but nothing still on our end

Nothing on my end has been restored either. I must be a loose end :)

Still having problems on homiemail-sub5 (had the same issues yesterday), no emails are coming in.

I have just picked up 120 messages, probably 90 SPAM, however, it looks like sub4 is a little better

It looks like my mail on homiemail-sub5 is working again, and webmail is working too.

I’ve been having email days for months now, not just today.

We are also having major issues with all our sites and databases being slow and causing errors. A support ticket has been submitted.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamObjects Maintenance – May 22nd through Friday May 31st (RESOLVED)
4:26 pm

DreamObjects Maintenance – May 22nd through Friday May 31st (RESOLVED)

Posted (May 20th, 2013 at 4:26 pm PST) by mir

Our Systems Engineering team will be performing maintenance on our DreamObjects service from Wednesday, May 22nd through Friday, May 31st. During this maintenance period you may intermittently experience degraded performance. The purpose of this maintenance is to implement key upgrades to Ceph, the underlying storage system which powers DreamObjects. As a result of the upgrades our development team will be able to provide some exciting new features, including CNAME support, in the near future. While we anticipate slower than normal speeds when storing or retrieving objects, please rest assured that your data is safe and will not be affected.

If you have any questions or concerns, please don’t hesitate contact support for assistance.

Update May 31st 4:20 PM PDT:  We are happy to report that the maintenance is now complete. As mentioned before, this upgrade allows for new and exciting features, like CNAME support. We are working on implementing these and more right now, and will be sharing the news with you soon through the Cloud segment of our Dreamhost.com and blog posts, so keep an eye out for them!

Update May 28th 1 PM PDT:  A quick update to let you know that our Systems Engineering team is continuing to scratch tasks off their clean up checklist. It’s still possible to see minimal delays during these efforts, but we’ve been very happy with the performance throughout the upgrade process. We expect to finish by the 31st and will continue to monitor and keep a close watch on the progress. We appreciate your patience!

Update May 22nd 11:20 PM PDT:  We are happy to report that all the upgrades ran without a hitch and are now complete! We are keeping this status post unresolved for now to clean up and continue monitoring the related systems. We will update this post again when the maintenance itself is finished. We appreciate your patience during this process!

This entry was posted on Monday, May 20th, 2013 at 4:26 pm and is filed under Announcement, DreamHost Features, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail connectivity issues on homie-sub3 and homie-sub4 ( RESOLVED )
2:24 pm

Mail connectivity issues on homie-sub3 and homie-sub4 ( RESOLVED )

Posted (May 20th, 2013 at 2:24 pm PST) by brians

Our Systems Engineering Team is currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for connection errors which customers on the impacted mail clusters are experiencing at the moment. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

UPDATE May 5th, 3:09pm PDT

The impacted mail service has been restored and you should now be able access the server again. Once more, we sincerely apologize for any inconvenience this has caused. If you are still experiencing service disruption please contact support!

UPDATE May 5th, 2:59pm PDT

The switch is back online and the network traffic appears to be back to normal but we are waiting for a final ‘all clear’ before resolving this matter. We thank you for your patience and understanding.

UPDATE May 5th, 2:46pm PDT

Our Systems Engineering team has tracked the problem to a switch losing power – they are correcting this now and we will update you further once it’s back up.

This entry was posted on Monday, May 20th, 2013 at 2:24 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

19 Responses to “Mail connectivity issues on homie-sub3 and homie-sub4 ( RESOLVED )”

May 20th, 2013 at 2:26 pmAndy Ringsmuth Says:

Good luck getting it sorted out! I know from personal experience that running mail servers is definitely NOT for the faint-of-heart. That which doesn’t kill you only makes you stronger, right?

May 20th, 2013 at 2:28 pmshoes4industry Says:

!!!!!! WTF what year is this 1998? come on!!!

Do you have an estimate until things get back online?

A happy monday.

I’m pretty sure I won’t be licking anything, DH…

Rock and Roll guys! Hope you get it rolling soon. Been a rough week or two here, seems like when it rains it pours.

@Andy Ringsmuth
If Dreamhost keeps dropping the ball I am likely to find out for myself.

I have an interview dependent on this email… this is very important guys :(

my account on homiemail-sub3 is back up now.

I am having clients call me already… You are hurting my business. Please provide a workaround asap.

Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

AND THEN WHAT???

Back up on homiemail-sub4…

That’s great Helen, but my client’s on the same server is not yet. Homiemail-sub3 right?

hi Wallknown, by their update, it looks like it’s not completely fixed yet.

@shoes4industry, I’m very sorry and our admin/syseng teams are working on fixing up the mail issues
@mark, please watch where you do that!
@up north, thanks for understanding
@Ben, we hope to have things back up and running shortly, a new update should be posted soon!

6 hours? what is a outage? the people in contact suport are working on that problem? why nobody answer?

i’m in homiemail master and my MySQL is down!! and NOBODY answer my messages!!!!

@fabricio, the switch is back up and we’re awaiting a “all clear” on the email situation on sub4, but your mail cluster is unaffected. We’ll have someone take a look into your tickets.

DYINGGGG WITHOUT MY MAIL! – -STILL NOT UP FOR ME….. ;[

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues affecting our US-West Data Center (Los Angeles) (RESOLVED)
10:01 am

Connectivity issues affecting our US-West Data Center (Los Angeles) (RESOLVED)

Posted (May 17th, 2013 at 10:01 am PST) by Oscar

We are currently experiencing some connectivity issues in our US-West Data Center (Los Angeles, CA). Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses. Any changes requested in the panel will be delayed throughout this outage. To see if you may be affected, please check your location in the DreamHost panel.

Update May 17th 11:25am PDT: All services are reporting back online now. Any new domain registrations attempted during this outage may have failed and will need to be re-attempted. We appreciate your patience while we resolved this outage. If you are having any issues with your sites, any changes made in the panel or domain registrations please  contact support, so we can assist you.

Update May 17th 11:25am PDT: Almost all services are reporting back online now. There are, however some delays will occur with changes made in the DreamHost panel. In addition any new domain registrations may fail at this time and will need to be re-attempted after that service is functioning again, we will keep you updated on that progress. We are still continuing to spot check all services to ensure they are working, but for the most part we’re in the clear. We appreciate your patience until we reach full resolution.

Update May 17th 11:00am PDT: Most services are reporting back online now. We are still continuing to spot check all services to ensure they are working, but for the most part we’re in the clear. We appreciate your patience until we reach full resolution.

Update May 17th 10:50am PDT: The cause of this outage is due to a hardware failure in one of our core routers. We have replaced that router and are checking all services to ensure they are online and working without any issues. We will continue to update this post as we have further information.

Update May 17th 10:30am PDT: This outage is currently affecting all web, mysql and mail services. Our network engineering team is continuing to investigate the cause and we will update this post once we have further information.

This entry was posted on Friday, May 17th, 2013 at 10:01 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

26 Responses to “Connectivity issues affecting our US-West Data Center (Los Angeles) (RESOLVED)”

Okkkkkkkkkkkkkkkkkkkkkkkkkkk

Is that why my email password change isn’t updating?

Again !!!!!!

PLEASE FIX!!
CAN”T SEND OR RECIEVE EMAILS_ BUSINESS SUFFERING!!!!!

Email hasn’t worked in over three days for several reasons, now this outage…
Had to postpone the launch of a site this week because of it.

this is a joke…and do you expect me to pay for a service like this ?…every day at the same time my websites are down…your service now really sucks…I am not renewing my contract for sure

change ur name into “nightmarehost”

even TUmblr is more efficient than you…and it is for free

I told you guys this was happening two days ago and I was dismissed.

@Max

Yes, The outage is causing a delay in panel updates.

@Dustin
@Randi

This outage is indeed impacting mail service. We sincerely apologize for any inconvenience this has caused. Our admins are diligently working towards resolving this matter

Will this going on FOREVER?!?!?

How interesting. This outage is preventing me from transferring my domain and hosting elsewhere…

@James

The outage is preventing panel changes from occurring. We hope to have everything fixed shortly. Again, we sincerely apologize for any inconvenience this has caused.

Is there a estimated time to solve the issue?
Will we receive discounts on montlhy fees for this problems?

@Leandro I guess no. And no. I contacted support about discounts/refunds and all I got was an automate reply.

FUCK YOU DREAMHOST! JUST FUCK YOU!

@Leandro
@Jess

We do not have an ETA at the moment. However, our admins are pretty close to a solution.

With regards to to credits for downtime, please submit a support request.

@Leandro Probably not. In fact, I received an email earlier this week that they’re raising the months fees.

@Steve

We still LOVE YOU, no matter what!!

Hahah, more one guys that today is friday!!! (gmt -03:00)

> To see if you may be affected, please check your location in the DreamHost panel

Can’t. My email isn’t working, so I can’t receive the email to authorize my IP address to check my location!

It’s 11:50 Pacific and I still am not getting emails. This is getting to be a habit with the LA server. About time for some new equipment and software!!

12:27pm and now I’m getting 500 Internal Service Errors. We’ve gone from interminably slow to broken on this ticket which is marked RESOLVED.

Over the years you have been a great and affordable service, but this is really beginning to effect my business. Please actually resolve ASAP. :-(

Still not working here in Houston at 12:48 pm PST…

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » VPS web service maintenance May 15th-18th — Extended till May 24th (RESOLVED)
4:58 pm

VPS web service maintenance May 15th-18th — Extended till May 24th (RESOLVED)

Posted (May 15th, 2013 at 4:58 pm PST) by andrea

We are conducting preventative maintenance on VPS web services starting 5pm PDT today. This is the first batch, and we’ll be going through all our VPS machines which will take us approximately three days, in segments.

Apologies on the late notice.  Since this fix was integral to stabilizing our servers after the brute force attacks earlier this week, we feel it’s necessary to roll out the changes immediately. This maintenance will require the web servers to be restarted after the change has been applied. The restart will cause minimal (few seconds) downtime per server.

We will keep updating this post with progress reports and monitor the status comments in case further action is needed on our end.

Update May 24rd 4:35 PM PDT: The upgrades are now complete! I am very sorry that it took longer than planned. If you run into any issues, please contact support so we can help you!

Update May 23rd 6:30 PM PDT:  We are in the finish line, only fraction of the machines remains. The reason we extended the maintenance by a week is to make sure that all updates are correctly done, and some ended up having to be redone – standard procedure especially in such a volume this maintenance was. As expected, this maintenance is not affecting customers other than the few second reboot times. We will resolve this status post as soon as the process complete. Thanks for your continued patience.

Update May 22nd 11:30 PM PDT:  Very sorry about the lack of updates on this status post. The maintenance is taking longer than expected, and we’ll update this post tomorrow with the new expected finish time. This process shouldn’t be (and reportedly isn’t) causing service issues other than the few second reboot time, but please contact support if you have any questions or concerns.

This entry was posted on Wednesday, May 15th, 2013 at 4:58 pm and is filed under System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

88 Responses to “VPS web service maintenance May 15th-18th — Extended till May 24th (RESOLVED)”

email is totally messed up today. I’m getting emails delivered from random times this morning some twelve hours later. During mid afternoon, was getting emails in realtime, but also getting emails from 6 hours earlier delivered. Its a mess. Is this more fallout from the homie mail server updates?

May 15th, 2013 at 7:11 pmprogress report Says:

Anytime now.

May 15th, 2013 at 8:03 pmLuther Lowry Says:

Couldn’t these changes take place in the middle of the night – some people require application uptime during evening hours – in my case, this is currently the PEAK time for my application to need to be available. What’s the point of having a VPS if it can go down at any time with zero notice!!!!!!

don’t sweat it just fix it

@Paul — The mail slowness is not related to the VPS web services at all. We are working on clearing out backed-up queues and making sure the mail is delivered promptly. If you have any questions (and you haven’t done so already) please submit a ticket: http://dhurl.org/dhsupport

@Luther — I’m very sorry! Unfortunately, our customers are world-wide so any time of day that we do maintenance will affect someone.

No worries, and thanks for the update.

Our site has been up and down throughout the day, but completely unreachable since 5:45 PM PDT. This doesn’t seem to match with the “few seconds downtime per server” mentioned above, so I’ve opened a ticket. Good luck!

@Eric – I am so sorry for the problems, and opening a ticket is perfect — that lets us dig into it. Thanks!!!

i agree with the mail problems. smtp appears to be down. i’m on good ol’ unreliable homie-6.
normally i just move on when there’s dh problems, but this time wasn’t good. i sent an email last night that didn’t get to my contact until this morning. then i had mail come in from yesterday, and earlier today tonight. help. please. this is not something i am doing wrong, nor is it my computer’s fault. i’ve been with dh since 1996. i’m not leaving. i just don’t want customer relation bullsh*t. thanks.

ps: webmail is down, as well now.

Not only every week… now, every day… problems, over problems with DH. The the most unreliable webhoster ever !

Where are all the progress reports? How do we know when our specific VPS’s have been updated? What is the nature of the updates?

@Lowry it’s ALWAYS an inconvenient time of the day somewhere. At least 5pm PDT is the start of the evening for those of you lucky enough to live in California ;-)

I opened a ticket 1 day and 2hrs ago, extreme urgent everthing broken…. No reply to date. What the hell is going on guys, all my websites have been down for quite some time now, very disappointed. No reply to any of the tickets I have opened. Getting pretty annoyed. Sort it OUT

announce says seconds, actually was down for 4 minutes. Yes it was the middle of the night, and normally I wouldn’t care, but the complete lack of prediction shown so far red-flag’s this. Why seconds expected and minutes actually taken?

@spence – the announcement should not have said “seconds”. It takes more than a few seconds to reboot any server and whoever wrote that should know better. You should also note that there’s no way to reliably predict EXACTLY how long these things will take. If they said seconds and it turned into an hour I would want to know what happened. If it turned out to be 2-3 minutes I would consider to be within the margin of error.

My sites are still unreachable. I’ve filed a ticket and still haven’t received any feedback. This really is bad :(. I’m going to have to migrate hosts for one of my projects if this isn’t fixed immediately. Do you all have an ETA? It’s ok to say no, but just be honest about it so that everyone can plan accordingly. I’ve been with you all for awhile and know your service is usually good. I would come back, but I need know what’s going on and what I can do to mitigate my issues and what expectations to have on this.

Please update with what you think is going on. This VPS fix didn’t fix the core issue I’ve been experiencing for days now.

Some of our sites have been down for 36 hours…at least it’s been that long since we’ve noticed. I’ve been with Dreamhost since 2001 and am usually happy with the service, but every time something goes wrong they seem to “circle the wagons” and provide unrealistic time-frames/expectations.

If we’re gonna be hosed for days, SAY SO. Some of us would move our sites someplace else until things are resolved at Dreamhost. I don’t want to(temporarily) host host internally, but WOULD HAVE started the process yesterday if I knew we’d be down for days.

How much you wanna bet, I’ll fuckin’ get out of here?? Unless your CSR calls me, reassure me, apologies and reimburse my last year’s complete services…

I run a small website and forum. Minimal traffic. They’ve been trying to pin my performance issues on me the last 3 days, had me go through the logs and block IPs, etc.. They send me form-mail responses with links to their wiki on how to improve site performance.. All the while their infrastructure has been blowing up.

Been with them since 2001, generally have been very happy. I’m done now. They’ve dropped the ball lately. Already started a plan on another service to test them out for a bit.

Have not been able to send mail for two days now.. ?

Why is no one responding to support? All of our sites have been down since early this morning. This is prime time.

PLEASE RESPOND TO SUPPORT TICKETS. OR AT LEAST TELL US WHAT IS GOING ON.

No email today. You’re making changes there too? Apple Mail doesn’t like your certificates because the domain is dreamhost.com, which doesn’t agree with my domain name, but even if I say “Connect,” it can’t.

Damn, I thought it was my end and I completely deleted my entire emailing list. When is this going to be resolved??

Apple mail is not working for me either. I cannot send any emails and it is very important that I be able to send them. Can I get an ETA on a fix or at least an update?

Also getting “The certificate for this server is invalid.” on Apple Mail.

Please respond with some updates. This is 21 hours since your post.

Mail service is intermittent. Frustrating!

I can get inbound mail (once I restarted Apple mail and continued past the “wrong cert” error), but can’t send outbound. A pointer to a fix (presumably how to get the new cert) would be welcome.

Another Apple user here… can receive but cannot send. I need to be able to send emails. Please send a remedy pertaining to the certificate!

Certificate problems as well on Mac. Do not want to log in but need my emails?????
Help!!

Was able to send a few emails so (fingers crossed) seems to be working now.

Cannot receive email through Apple Mail but can send.

I’m on Elbert, and I get a “The certificate for this server is invalid.” message when attempting to send email.

Great, at least3 days of BAD SERVICE, i want to migrate to another provider, goodbye dreamhost

YEP, my Apple mail is NOT WORKING either.

For those of you experiencing issues with Apple Mail, please refer to the related status post: http://bit.ly/176lC2I

“Update May 16th 10:15am PDT: The changes have now been applied to all of our mail servers. If you have any issues connecting to your email account within your email client or sending/receiving please make sure you make an exception for our new certificate. Once you have done so, you should have no issues connecting or sending/receiving. In some cases you may need to delete our old certificate in your email client and accept our new one and you should be all set.”

Please try accepting the new certificate; In some cases you will need to delete the old certificate. If you still run into trouble after that, please Contact Support directly via the panel with details on what you’re seeing. We’ll be happy to investigate! Thanks!

My site has been down for at least 24 hours and probably longer. Just get a blank screen every time I try to load a page. Wrote DH yesterday, and someone wrote back that everything is fine. It most certainly is not. My follow-up email this morning went unreturned. This is the worst downtime I’ve ever had with DH, and lousy service to boot.

Are you kidding me?
A few seconds of downtime?
My sites have been down for hours today.
HOURS.

I’ve sent several messages to you and no answer! My blog has fallen more than 30 times just today. Help me!

Hi Dreamhost team,

Our server has been down and websites unreachable since yesterday. It has been more than 24 hours since it went offline. Can you please tell us how many “few seconds” more this is going to take. And let us know what dictionary you are using because I’m quite confused of what the word few means already.

Thanks!

Guys this is the second time in 4 days all our websites have been inaccessible for over an hour, and at least the 10th time this year already. Need some reassurances on your service else I’m going to have to look to migrate, this is getting unacceptable.

Since the e-mail outage, I am receiving e-mail, but I am unable to SEND e-mail.

Please advise.

Thx,
David

It would be really great if you guys could actually be a transparent about the issues, and not post these ridiculously optimistic and unrealistic notes about ongoing issues. I haven’t been able to get mail on mobile for over 24 hours, and I get constant secure connection issues with your servers. It’s pretty clear you guys have massive problems going on.

-rt

All of our sites have been down for over 24hrs I opened a ticket yesterday morning and have had no response. PLEASE HELP!

After a few days of on and off service due to the attacks everything was back to normal: then for the last 16 hours I was just getting 503s. Bur I could use the Dreamhost panel, ftp and phpmyadmin so all parts seemed to be functioning. Changed .htaccess file, disabled all plugins etc. through ftp: nothing.

Then I found this thread in wordpress.org which suggested turning off Extra Web Security in the Dreamhost panel ( Manage Domains/[site]edit ) under the Php Mode section.

https://wordpress.org/support/topic/error-service-temporarily-unavailable?replies=8

Everything now works. Hope this helps someone.

How’s this maintenance going?

dreamhost website down ?????

Site down again. Everyday it’s a new chapter of the down/up websites-hosted-by-dreamhost. Seriously, how this can happened for six months? Slow, down. Everyday a new problem.

Site is down yet again. Just another day at Dreamhost. I think this is the last straw. Time to migrate to a new more reliable service.

I changed the Apache web server nginx. It’s all out of breath. I went back to apache but still off the air.

Down again. It says few seconds but it’s hours.

Its down again… Guys tell me one thing .. Is it true that only shared hosting is getting affected ? If i upgrade to VPS will it help ?

I am canceled both my VPS :), will keep the web hosting.

Actually, the fix above of turning off Extra Web Security stopped working after a bit, and the site is down again. But at least that’s one thing accounted for.

Llevo varios dias practicamente sin cocexion en nuestra web, y cuando va la velocidad es muy lenta, me parece una verguenza este servidor.

Took several days without Connection practically on our website, and when you speed is very slow, it seems a shame this server. Solution to the problem yaaa!!

Why is DH always down? Seriously? Also why is my wordpress blog down but not my website???

You know what, I understand if you are having serious problems, or attackers but this is ridick. All I wanted was a reliable website, and you say this period will end tomorrow. If not, I will leave.

This update keeps the server slow?

my database is down again. We’re done with DH!

Now my database is down again too.

any time frame?

6 years with DreamHost: my site is down for hours EVERY WEEK now.. for $70 per month on a VPS!

Migrating to either HostGator or SmallOrange in the next few days….
Any recommendations from other frustrated users?

DreamHost to me is just now an utterly useless and damaged hoster.

Site has been very slow for days. this is just one more problem in a list. Mail was very slow at first, but seems to be better, now my site is extremely slow to the point of users just clicking away from it. Not acceptable when I am trying to host some information to others.

Hi guys,
This is really going out of proportions. my website and my mail isn’t working for days now. Please do something!!!!

Way, way out of proportion. The monthly emails that attempt some humor are not doing very well when the entire company has a serious problem with delivering the service they promise. That humor just makes me think that nobody is really watching the store and it’s just a big game to them.

SMTP – Mail down… says it’s “offline” — What do I do? When is this going to be fixed???

Dear DH I don’t appear to have been affected by this maintenance (yet), but this is 5 days into a 3 day planned maintenance, and as far as I can see you’ve not posted ANY progress update ? No news; good news or bad news ?

May 20th, 2013 at 7:40 amShelley Johnson Says:

Mail down, sites down, DB down… WAAAYYYY too many times recently. After 7 years, DH = history.

We have several sites hosted on Dreamhost and we cannot send emails from any of our accounts! This has been going on since last week! We’re trying to run a business here! How have there been no updates on the status of the since May 15th? WTF? Losing patience here.

Site down, DB down after sunday 18;00 :(

May 20th, 2013 at 9:34 amH. Guerrero Says:

Whats happen in DH?
Who i can change of web hosting? It’s possible? I have a hosting plan of 3 years…

My site is so slow and barley responsive! I opened up a ton of tickets and havent heard anything! HELP! Please… These are clients.

My site is timing out! cMON! HELP US! tell us something! Database connections are failing!

can’t even get on to dreamhost site – “bad gateway”.

no email on one computer for 4 days yet ok on another.

(no, i have not changed account info.)

We are also experiencing downtime on our site for over 3 hours today and several last Friday. We’ve haven’t received a response from the new ticket that we opened today several hours ago. Dreamhost, please give us an update! This blog post say 15th through 18th, please give us a status update for May 20th!

Can we even get an update on what’s happening, and when it MIGHT be fixed?

From Dreamhost: “We will keep updating this post with progress reports and monitor the status comments in case further action is needed on our end.” -Posted (May 15th, 2013 at 4:58 pm PST) by andrea

Yeah, right! At the very least, let your customers know what’s going on! No updates since this initial post! So frustrated!

we’re down also……..UGH

E-mail has been screwy for day, but just got screwier. Had been able to receive via my iPhone and webmail, but not Mac Mail. Tech support has me trying all sorts of fixes via Mac MAil, none of which worked, and now, suddenly, iPhone and webmail are no longer working, either. Luckily, my website is still up but things seem to be getting worse instead of better, Come on Dreamhost, if I wanted this BS, I would have stayed with Bluehost.

E-mail is gone…

E-Mail has definitely been a bit rubbish the past few days. It’s broken right now.

Last time I contacted support and told them I was using pop3 account their reply was basically that I should use Imap instead. Well I’m sorry, but Dreamhost IS supposed to support Imap AND Pop equally as well. Saying I should use something else is NOT a good answer. Just fix the damn pop3!!!

Email has been offline sporadically for four days and finally stopped working for all 7 of our users. We keep getting the MacMail error message that our passwords or other data may be incorrect; however it was working earlier today and no one has changed their preferences. Webmail isn’t allowing me to log in either, so from a communications/customer service standpoint, this day is stacking up to be kind of a disaster.

My site is down after 1 day!!!! What happen DH???

All my sites are down, I’m not sure how long they’ve been down. My clients are calling me, when will this be fixed?

Another day with problem ,
It’s nice that I get to say it is the fault memory.

ITS MAY 21 .. IS THIS STILL HAPPENING??? (POST SAYS COMPLETE BY MAY 18) … also no additional status reports???

And we are down AGAIN. Every single day my site is down now.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency software maintenance in our US West (Los Angeles) Data Center May 15th, 8pm (4 hrs) (RESOLVED)
2:43 pm

Emergency software maintenance in our US West (Los Angeles) Data Center May 15th, 8pm (4 hrs) (RESOLVED)

Posted (May 15th, 2013 at 2:43 pm PST) by andrea

In light of what happened earlier today in our US West (Los Angeles) Data Center, we are going to perform an emergency maintenance tonight to help further solidify our network from this type of incident. We are still looking into the exact root cause of the incident – keep on a lookout for a detailed status post about it tomorrow.

During tonight’s emergency maintenance, we will be adding a patch to the core router software.

As our CEO mentioned in an earlier status post comment, we have been continuously upgrading both our hardware and software in all our data centers. We understand, and also experience, the frustration these continued situations are causing. We are working closely with the vendors as we progress through these network upgrades and changes.

We are estimating four hours for this maintenance, to adequately pre-test and make changes and test again. We don’t expect this procedure to cause any noticeable service interruptions. Once we have started, we will be monitoring your status post comments to make sure everything is going as planned. We will update this status post as soon as the maintenance commences and will be updating it with progress reports.
To see if you’re affected by this maintenance, please check your location in the DreamHost panel.

Update May 14th 11:20pm PDT: That went perfectly as well, so we’re closing this one for good!

Update May 14th 10:45pm PDT: Since we have a sliver of time left on this maintenance, we decided to update one more aspect of our core systems. So, we’re re-opening to update a piece of hardware we have. We expect minimal impact.

Update May 14th 9:50pm PDT: All networks are moved over, so that concludes our maintenance! Thankfully, no downtime was experienced! We appreciate everyone’s patience on this.

Update May 14th 9:37pm PDT: Humming right along now, still haven’t seen any issues, and we’re almost done!

Update May 14th 9:14pm PDT: These al go by pretty quickly, so we’ll update if we notice problems. We’re doing a list of shared web, mysql and VPS traffic now. I’ll update when the remaining networks are moved

Update May 14th 9:10pm PDT: Now we’re doing a network that comprises almost exclusively shared web services. We don’t anticipate problems at all

Update May 14th 9:01pm PDT: That went perfect as well! The next two networks are internal, and won’t be customer facing at all

Update May 14th 8:55pm PDT: We see no complications form that last update, so we’re moving on to the next. This one compromises VPS services, primarily. Again, we don’t expect any downtime to happen.

Update May 14th 8:50pm PDT: That went perfectly! Now we’re moving on to the next network. This one contains shared web and VPS traffic. We don’t anticipate any downtime.

Update May 14th 8:40pm PDT: Internal network is now moved without incident so we’re moving on to another network. This network contains some of cafe commerce ecommerce networks, as well as our central services (webmail, dreamhost.com, panel.dreamhost.com). We don’t anticipate downtime, but are on the lookout.

Update May 14th 8:21pm PDT: Ok, we’re starting. First, we’re updating an internal network, so no customer facing service should be affected

Update May 14th 7:59pm PDT: We’re gearing up to start this maintenance! We’ll update as we get more info.

This entry was posted on Wednesday, May 15th, 2013 at 2:43 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

8 Responses to “Emergency software maintenance in our US West (Los Angeles) Data Center May 15th, 8pm (4 hrs) (RESOLVED)”

Something’s getting cached somewhere, so it’s not quite transparent. I’m seeing ancient versions of pages… ssh into the server shows the latest version, wget/browser loads the old one. That’s on bucks; pagespeed/akamai stuff is off.

Seems fixed now. Damn gremlins. :-) Good luck with the rest.

@Tom Lin, that was before we did anything, so, not related to our maintenance =)

I’ve lost mail service it seems. I am at the L.A. data center, on homiemail-sub4.

Is this related to this maintenance effort?

homiemail-sub4 is down

@Dan, hard to tell, as you aren’t specific as to exactly what you’ve lost. We’re completely done, so, I don’t believe its related. Submit a support request, and we’ll be happy to look into it for you!

I’m on homiemail-sub5, have received mail throughout the day but haven’t been able to send anything since last night.

@drue, that would be unrelated as well, submit a support ticket, and we’ll look into it! Make sure you include your outgoing mail settings for us.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues affecting subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)
10:02 am

Connectivity issues affecting subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 15th, 2013 at 10:02 am PST) by Justin@DH

Some of our customer in our US-West Data Center (Los Angeles, CA) may be having some connectivity issues at the moment. This includes web, mysql services, and specifically the homiemail-sub5 mail services (You can find this in the panel via Account Status on the top right, and checking “Your Email Cluster”).
To check if you’re in the affected data center, please check your location in the DreamHost panel.

We are investigating the root cause and are working towards resolving it. We are very sorry for the inconvenience this is causing, and will post update as soon as we have more information.

Update May 15th 12:45pm PDT: We believe the connectivity issues to be resolved at this time. There will need to be a network maintenance tonight, and a new post will be up soon with details. Again our apologies for the downtime caused by this and your patience as we complete network upgrades and hardening.

Update May 15th 11:50am PDT: We are continuing to restore services for the last few remaining servers that are reporting. An an update with more details will be forthcoming and provide details of a future network maintenance. Thank you for your patience while we restore the remaining services.

Update May 15th 11:30am PDT: At this time services should be restored for nearly all customers, and we continue to investigate and monitor the situation. An an update with more details will be forthcoming and provide details of a future network maintenance. Thank you for your patience while we restore the remaining services.

Update May 15th 11:10am PDT: We are continuing to see web, mysql and mail services recover in the Los Angeles (US-West) datacenter. We are continuing our tests and checks of our monitoring system to get any remaining services up and running. Thank you for your patience. We hope to have all services restored soon.

Update May 15th 10:50am PDT: We are seeing web, mysql and mail services recovering in the US West (Los Angeles) Data Center at this time. We are still investigating and making sure that all services and networks are working properly. We hope to have all services restored soon.

Update May 15th 10:33am PDT: Unfortunately, at this time, all services located at the US West (Los Angeles) Data Center are down. To check if you’re in the affected data center, please check your location in the DreamHost panel.
Our network, admin and datacenter teams are working hard to investigate and resolve this issue. We hope to have all of those services restored again soon.

Update May 15th 10:19am PDT: We are continuing our investigation and work towards resolving this issue. We are seeing many services restored, including the homiemail-sub5 problem, but there still remain some specific network blocks that need additional attention.

This entry was posted on Wednesday, May 15th, 2013 at 10:02 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

172 Responses to “Connectivity issues affecting subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)”

I think we know the cause. Poor service.

I’m on the East Coast and my sites are down. Again. This is pretty bad, DH.

7 days in a row of disrupted service

awesome

Welcome to your regular afternoon “outage” .. do you want popcorn with that?

Down RIGHT as I send a link to someone. Please hurry!

All of my websites are down again for the 3rd day in a row. I understand the issue with DDOS attacks or whatever is going on on your back end, but I in reality, I shouldnt have to care about what you have to do on the back end to provide me the service that you guarantee upon sign up. Do what you need to do and get this problem fixed once and for all.

here we go again… third day down.

Is a shame…

Connectivity issues? Another phrase for completely broken again?

5 DAYS, 5 DAYS, EVERYDAY AT THE SAME TIME. EVERYDAY. YOU DON’T DO ANYTHING.

I have business and you don’t seem to care about them at all.

If tomorrow there’s the same problem, i’ll send the dispute to paypal so i can retrieve my money back. This is a big scam.

completely down. again. Not just connectivity but no ssh access. I set up monitors to start tracking uptime, because this is becoming a problem.

Literally hours before i have to show a client and this happens again

Really

I have been with you for 6 years and never had it this bad before.

AAARGH

Agree with Mike totally. I understand what is going on. But it isn’t our concern. Our concern is that our sites are up and running. We pay you to make sure that happens. And right now, over the past several days, we’re just throwing money away because of Dreamhost’s inability to provide stable service.

My sites were down briefly (4 minutes, that I noticed) and are back up now. Thanks for the prompt resolution dreamhost team.

EAST having problems as well

Usually this is fine, but I have updates to make on my websites today. Please return as soon as possible.

3 days in a row, not bad guys :)

Sorry for the issues related to this outage. We have our staff working hard on the issue to have it resolved as soon as possible.

I’m on the East coast, my e-mail is down again! please, at what time I can use this?

Again? Seriously!?!?

i told them 3 hours ago their were issues and they now seem to be getting around to posting and looking into. typical

I’ve rarely had site issues, to counter what those have stated in this thread…but, that said, I cannot afford this downtime. At all. My yearly renew is next month, if this isn’t resolved within 30 minutes, OR it happens again in the next 30 days…hosting is easy to come by, so, ’nuff said.

Cannot get into Control panel. Email ip authorization not sent by DH.

DH really sucks… every week “problems”!
My hint: try a other hoster and you will smile.

These down-times were a great motivation for me to set up redundant hosting with DNS failover (at dns made easy) and automatic SQL syncing. So my important sites are up, just not at DH.

Your physical location has NOTHING to do with your data center location, people.

Hey, I’m on the east coast and mine are down too…from the comments above, you really ought to update this to clarify it’s not just affecting the West Coast, since that’s clearly inaccurate, no?

Day 3 of outages?

Yeah, I’m done. As soon as I can get the remainder of my services migrated elsewhere, I’ll be closing my account.

I’ve been with DH for a lot of years, but this is too much.

“MAY be issues”??? Are you kidding me?! Our website has been on and off for the last two hours and is now completely down again. What’s the excuse THIS time, Dreamhost?

OMG… DH sucks

I could set my clock to the daily noontime site outage.

East coast, here. All sites down again, like yesterday. I didn’t mind yesterday’s downtime, I was going to make up for it today… but I can’t. And a website is due for delivery today. This is the first time in appx. a decade that DH’s downtime has begun affecting my business. :-(

AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!!

AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!! AWFUL!!!!

May 15th, 2013 at 10:20 amDreamhost Lies Says:

Just wait, the next update will be that it’s not just affecting a “subset” of users, and in fact, a HUGE chunk of customers are experiencing all sorts of downtime!

Like every other freaking week these days! As soon as this comes back online, I’m migrating all my sites away.
Good riddance, nothing but headaches at Dreamhost!

Issues happen, I get it. Your primary business function is to provide a stable hosting platform but you are consistently failing at this. It feels like your team sits around and waits for things to happen instead of proactively using the subscription cashflow that we provide to seek new security measures to stay one step ahead. My best option is to now move away from shared to a vps but I absolutely will not be buying that service from DH. You have lost my confidence just like my clients who have now lost their confidence in the hosting service that I provide for them. Embarrassing, isn’t it? I’ll be contacting you in regards to the credits to my account that you will provide for not living up to your service promises.

Maybe DH should spend less money on cutesy stuff around their office, and less traveling to meet customers, and more money on their servers, pipes, and staff. You know, like everyone else. Everyone else who is up consistently. DreamHost has the stability of Lindsay Lohan, and it’s ridiculous. DH doesn’t offer any monetary compensation for the constant downtime they have so I believe it’s time for me to take my business elsewhere. It’s not like they really care though, they have already proven they don’t care about us customers. As long as they market their brand, they’re happy. Hopefully I’ll find a new host today, I don’t mind paying more as long as they have a stable network.

Fuck man. You guys a just screwing up 6 months of work prior to a Web Service launch yesterday, right in the middle of your attacks of 10h45 am… All failed because of you.

JB, I have a VPS through DH, and it’s down just as much as shared hosting. Nobody waste your money, seriously.

My sites went down about two hours ago, went back up for 20 mins, and are back down.

Holy Fuck, I have work to do. I should start taking the afternoons off.

100% uptime. LOL LOL LOL

maybe 50% is more like it

and… I just got an email saying that Dreamhost found no problems reaching my site, yet I can’t get to it and remote testing says it is completely down. Jeez, guys, this is not that hard…

I’ve been with DH for 6 years now and have never complained when things go down. Occasionally, sh!t happens. However this is the third day in a row, and it is really becoming an issue.

What’s the deal?

LeftOverCake.com stated it before, but it obviously needs stated again. Your physical location has nothing to do with your data center location. Your sites may very well be in the US-West datacenter, even though you’re on the east coast, or Poland, or wherever.

Let me reiterate… if you LIVE on the east coast that does not mean your SITE is hosted in the East Coast. If your site is down, odds are you’re on the West.

@tyler, thank you!!! Haha. Finally somebody who understands.

CEO OF DH

NOW I REALLY HATE YOU, YOU MAKE MY LIFE MISERABLE AND YOU EMBARASSED ME INFRONT OF MY CLIENTS, ALL ARE CALLING ME WHOM I RECOMMENDED DH YEARS AGO.

WHAT A SHAME!!!

YOU SHOULD REGISN AND LET SOMEONE ELSE TO RUN THIS COMPNAY.

WHAT A PATHETHIC SERVICES…

ENOUGH IS ENOUGH

WHAT THE HELL IS WRONG WITH YOUR DATA CENTERS, THOUSANDS OF PEOPLE LEFT YOU AND MORE LEAVING YOU, IF YOU REALLY WANT TO CLOSE THE SCRAP COMPANY THEN GO AHEAD AND NOTIFY US, SO WE GO TO OTHER HOSTING PROVIDERS.

I AM LOYAL CLIENT OF YOU SINCE 5 YEARS. NOW I FEEL I MUST LEAVE YOU GUYS.AS YOU ARE NOT PREDICTIBLE.

YOU ARE AN NIGHTMARE HOST.

JUST I HAVE THREE WORDS FOR YOU

SHAME
SHAME
SHAME

the ceo should be fired for sure. the company was much more stable before he came on

@EVERYONE

We do apologize the for the problems that you have experienced. Please understand that we are diligently working towards stabilizing our service. The investigation on this latest outage is ongoing and we will continue to update you are information is made available. Thanks again for your continued patience.

Well. My site is about preventative mastectomies to prevent breast cancer and a story about me and my site is supposed to air on Inside Edition in 3 hours. Perfect timing for this. It seems like this happens once a year, and it’s getting really old. This news story could make a huge difference for me, and now this opportunity may be lost because my site is down. Each year I suggest mirroring options, but they are never addressed. Such a disappointment, I’ve worked for almost 4 years to get this chance on TV, and when I finally get it, poof! No site.

I will be moving to either inmotionhosting.com or mediatemple.net.

You should check them out.

Later

Database Error: Unable to connect to the database:Could not connect to MySQL

when I first had monitoring alerts go off, the cms sites (joomla, wordpress) were down, but static were up. Then everything went down. Then everything went up, then CMS down…

Just checked… everything is up again…

I’m not about to jump ship… that sounds like a lot of work to migrate, and a lot of data to transfer…

However, if another hosting company had a script that ported everything over server side and made it one-click… I would be considering it…

Hopefully Dreamhost will fix this and get back on track. I’ve been happy with service provided, support given… all until this attack on them and the intermittent service.

Was fixed and now broke again :-/

AHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Anybody use the Google Apps service for email? Gmail has gone down on me before, and it would be great to have them as the backbone instead of what seems like a homemade server network.

its funny, it says to click Location to be taken to the panel to check and see if you are effected. I assume by the fact that the panel is down I’m effected. Oh, wait, I already knew I was effected because ALL MY FUCKING WEBSITES ARE DOWN FOR THE THIRD DAY IN A ROW!

We can’t send out email.

and now all down again….. #$%&!!!!!!!

Service to my web site and SQL database has been spotty since before 9am PDT. I had a window of good service starting about 10:15, but it just went south again. My site is on homiemail-sub4, not homiemail-sub5. I will be patient.

homiemail-sub3 email been slow and often not working at all for hours now. Mostly with email sent to within the same domain. Mail from outside the domain gets through better than from within the same domain.

everything is down, website, sql, email, panel

WTFF?

Thank you Dreamhost, every second we lost money.

@Steve
@BRAIN

YOU ARE RIGHT STEVE,AFTER NEW MANAGEMENT ALL IS CRAPY SERVICES. BEFORE IT WAS SO AWSOME I BELEIVE THEY TOOK CRAPY DATA CENTERS, OR OUTSOURCED ALL DATA CENTERS TO A COUNTRY LIKE EXISTS ON MOON.

EVEN MOST PATHETHIC AND POOOR AND TECHNICALLY LIVING IN CAVES THIRD WORLD COUNTRIES HAVE MORE DECEENT SERVICES THAN YOU.

YOU CLAIM THAT YOUR ALL SERVERS ARE LOCATED IN USA AND ALL SUPPORT IN USA, I DONT BELIVE THAT USA HAS SUCH CRAPY SERVERS.

SO POINT IS CEO OF DH MUST RESIGN IF HE HAS ANY SHAME AND HUMANITY.

WE NEED PUBLIC APOLOGY FROM HIM LEAST.

This is happening more and more.. i might have to start looking for a new host. Whenever i try to update my site it seems to go down again.

What can be done to prevent this from happening so often? I agree that there is great value in what you provide, but this has been very inconvenient the past week. Am evaluating moving a new client to Dreamhost who has many sites. Should I consider Dreamhost stable enough?

I am in New York. Site is down. Email is down. ENOUGH ALREADY!!! you are really pushing it!

People are dying here!

/**
Going to cast some OpenQNL
out into this windstorm and see if we can make some headway.
Stand back everyone.

http://github.com/EarlyClues/OpenQNL
*/

function resurrectWebsite() {
if website.deadInTheWater? {
website.revive!;
}

//log
if website.canWalkOnWater? {
sendMiracleEmailReport(“staff@earlyclues.com”);
}
}

function addDefensiveShieldsForSpiritualWarfare() {
while (website.down?) {
printf “By the Power Of The One Most Magnificent I Command Thee To End These Attacks!”
}
}

addDefensiveShieldsForSpiritualWarfare();
resurrectWebsite();

everyone needs to tweet and social media about this and tag them in it as the only way for them to take this seriously is for it to be out there in the public so everyone knows that this happens all the time and who all it effects

There are no words for the amount of damage this is doing to my company today. TODAY OF ALL DAYS!?!?!? REALLY?!?!?!!?

On a side note, has anyone ever tried to hold them to their uptime promise where they take money off of your bill for downtime? I called about it after being down for a few days around a year ago. They pretty much laughed in my face. I guess I should have left then.

My sites were up for a little while but are down again as of 1:37pm EST. :(

Also I was beheaded by King Joffrey.

Well I have been having issues sense this morning when I raised a ticket to you guys.

Also why do you have Cloudflare IF the caching of the site doesn’t work properly? Please tell me this because if it worked then my site WOULD be able to be viewed as there would be cache on the Cloudflare servers.

Damn it! I just got done clearing my inbox… website explodes… inbox is full again… why do you hate us DH… Why do you hate my inbox being clear!?!?111oneoneone

May 15th, 2013 at 10:39 amPhilip Owens Says:

I’ve never had so many problems with any other hosting company. This is totally insane. You send out the rah-rah newsletter every month expect us to forget that multiple times a month your services go down. Get it together or I’m headed back to Verio, who I’ve been with for fifteen years and the only outage I ever experienced was when Hurricane Wilma walloped their data centre in Florida. That’s a pretty good excuse I reckon for an outage.

I would *love* to file a ticket… unfortunately it’s impossible as everything is broken… yet again. I’m getting fed up with this bullshit

Well I can’t check if my website is on the affected data center since the Dreamhost panel itself is not reachable, how convenient.
When my website is up, it’s slow as hell…

Im a client since 2006 and am considering looking for another more reliable web host, unfortunately.

All our Sites and Services Down. =(

note to self, on paper, since my applications on dreamhost are unavailable…
[ ] look for stable hosting company
[ ] backup existing sites immediately to minimize work on moving

three years, and this is just getting really really bad.

JUST STAY AWAY FROM DH.
i already knew that they are not stable years ago but I was too lazy to move my files and just let it stay there. I should have spend the extra time to move out.
DH = never again.
they just give you headache ever once in a while. Worst hosting

I get it that people are upset, but complaining like children isn’t going to get the problem resolved any quicker. If you need more consistency, find another hosting company and pay for it.

I’m rooting for you DH support – for a speedy and hopefully long lasting resolution!

MISSION CRITICAL CLIENT DELIVERY TODAY

WTF WTF WTF WTF WTF

DAMN YOU DH

DAMN

YOU

Down again. Central TX.

May 15th, 2013 at 10:41 amHernan Cortes Says:

I´m in Spain and have problems with my webs too.

May 15th, 2013 at 10:41 amMike Mathews Says:

Wow, you guys are hard core.

Anyway;
DH, would be nice to know what the cause is once it’s figured out. I’d like to know how you’re handling it as well.
We have been having spotty service issues recently and I’ve been trying everything to optimize our database as well as our load times to try and speed things up.
I’m starting to wonder if my issues weren’t server related from the get go.

Updates are appreciated.

My sites are ALL down and no mail either. I’m connecting via the US west coast. I can’t even pull up the map referenced above. I can’t even connect to anything.dreamhost.com. I can only connect to dreamhoststatus.com

Really a bad day for this to happen! I hope dreamhost gets me dreaming again soon!

Yes I know it’s the wordpress attacks but it’s never been as bad as it is now.

This is really bad. I’ve been with DreamHost since 2005 and it didn’t use to fail constantly like it has been recently. My webmail goes down, sites are unreachable. Just really bad service. I have even recommended the service to others. Now, this makes me look bad.

I will need to research the LoE to move my sites with their databases (which some are large) to another hosting company. I’m tired of dealing with this constantly.

May 15th, 2013 at 10:42 amAndrew Bywater Says:

ALL MY SITES ARE DOWN!!!!!!!!!!!!!!!!!!!!!!!! HELP!!!!!

looks like the CEO is chilling at an event as Rome burns

https://twitter.com/DreamHostSimon

As much as I hate using godaddy’s backend, I have never (that’s right, never) had an outage with them in the last 6 years with sites on their service. The same cannot be said for dreamhost (I’ve probably had an account here 3-4 years). Seems like every few months there is a problem. This recent one has by far been the worst and longest lasting spanning days.

super classy.

THIS IS SORRY AS HELL! Im leaving this crap hole as soon as possible!!!

BTW, I’m in Brazil.

May 15th, 2013 at 10:44 amTravis Snoozy Says:

+1 to dreamhost panel not reachable. Mail is down (how I noticed this outage), so I can’t send a support mail that way. It’s almost not worth filing a ticket, because by the time I can, the outage will most likely be over (and the measly one-day-per-hour credits don’t start piling on until after you’ve filed the ticket).

Ditto John. Just as soon as I can.

Can i get my install moved to whatever server dreamhoststatus.com is on?

@Steve – That’s what CEO’s do…. you don’t think they actually work for a living?!? He’s probably giving himself a raise right now…

Sites were down, email down, dreamhost panel down, just back up… hopefully fixed. I am on the East Coast… why am I going through that data center?

This is not “complaining like children”. Yeah stuff happens, but if it goes on day-after-day-after-day then you have to start looking for who is not doing the job they are being paid for.

In this case I pay to have my webservices available 24x7x365, and once, ok, I take the hit, but day after day for several hours a day – that’s no longer acceptable. I don’t hear Dreamhost acknowledging any internal issues, it’s always DOS, or external forces. Yet I know the response time for my websites is consistently getting slower, with no changes in content…

Can we get a refund?

Can someone lay out exactly what is necessary to change hosting? Transferring all files and databases, get everything connected, that’s all fine, but domain names would also have to be transferred, right? Because it is not just the hosting that is effected with these outages, right? I seem to remember that the DNS info for domains registered at dreamhost is also effected. So just getting a new hosting account and pointing domains to the new host wouldn’t solve the problem because without transferring domains the DNS still goes through dreamhost, right? Gosh, what an expensive and times consuming pain in the ass.

C’mon…. REALLY???? My clients are waiting!!!!!!!!!!!

I’m up again. Thank you, DreamHost!

May 15th, 2013 at 10:46 amAndrew Bywater Says:

What makes me really upset – is that I “chatted” with a customer service rep that said ALL THEIR SERVICES are live…and that it’s an issue with my wordpress website – that I need to install a caching plugin! If this has been going on for multiple days… how do their own people not even know they’re having a problem!!!

No! not again homiemail-sub5!

Solve this asap!

It os not acceptable this as a recurrent trouble!

Drear Dreamhost,

I would recommend you host the panel application with a provider whose systems are up all the time so that we can at least file problems and request support. As it currently stands, your support site is as reliable as your hosting service (i.e., not at all).

On that note, hosting with another provider may also make you aware of problems with your network sooner than is currently the case. Based on those weekly outages, it seems that it takes you an hour or more to acknowledge problems with your systems. Also, you somehow seem incapable of providing updates more than once an hour, which is rather puzzling.

Lastly, would it kill you to proivide phone support without having to nickel and dime? I understand that it costs money but it would allow you to provide a more effective customer support when you systems are not working as advertised.

Thank you,

A very unsatisfied customer

I wonder what host DH site (panel and status) is using. DH site seems to be up all the time.
Terrible Service DH!!!

Can I go play golf now??

back up… seems slower, but operational.

@EVERYONE

We really do understand your concerns and frustrations. Our admins are continuing to work towards stabilizing service. Thanks again for your patience.

Andrew Bywate had the same experience where they act like you are crazy and everything works than 6 hours later they say o look there is an issue we are on it

It could be a lot of things

It could be a DDoS, it could be the OpenVZ (Virtualisation) exploits, it could be……..

This is just absolutely unacceptable. Even the dreamhost.com site is unpingable. I’ve had an account with dreamhost for over 4 years. I’ve recommended the company to all of my clients and developer friends. But I have never ever ever experienced downtime like this. I gave you my patience for the last few outages.

Sad to say, because I really did like Dreamhost, but I have no more patience left. This just should not happen.

Sites are up/down/up/down/up/down, and now the DH panel is down as well. What is going on?

Everything just went down again .. thought it was better.

hosted in US-West and my sites up&down, again.. really..

as I write, seems to work..

a lot of years with your Company, and this never happened before, do something, guys

Ouch, sites down again :(

I’m done, i do everything i can to pay your invoices and your are making it hard for me to get the money from my clients + you never provide any compensation for the poor quality of your services. As soon as I can get the remainder of my services migrated elsewhere, I’ll be closing my account as soon as possible , im fed up.
I’ve been with DH for a lot of years, but this is too much.

Sites back up (for me) for now.

So…me and your support team will do lunch again tomorrow as I continue pulling my freakin’ hair out and get an ulcer as my inbox (thankfully, not hosted with DH) stacks up with emails from angry clients that are starting to notice the lunchtime outage trend too?!?!?!

I’m freakin’ tired of making excuses for you while day after day…you make us look worse and worse.

WTF?! It’s going to cost thousands to transfer all of my clients email accounts off of DH and on to a REAL hosting provider.

Who at DreamHost do I bill for my loses????

this is effed up. i’ve been a customer for many, many years. you guys need to do something drastic at this point. i’ve accepted the patchy service over the years. it’s getting more frequent and the duration of downtime is increasing. if this were a free service i wound’t say a word. i think a good analogy of this situation is if you went to a decent restaurant, ordered food and sat around waiting… only to receive excuses about why you didn’t get the food you paid for… without getting the food, just excuses. we’re paying you to host websites and web applications, administrate the servers, provide security for dos attacks, etc. at the rate things are going i can go to bestbuy, buy a $200 computer, hook at up at my house and host my own content. i’ll probably get better uptime… fire someone, hire someone, buy new servers, get a better internet connection… i don’t care. just do something. thanks.

Ryan,

I assume you are billing your clients far more than you are paying DH

GET YOUR ACT TOGETHER DREAMHOST! I’ll no longer recommend DH when for the 3rd day in a row my site is down. And this isn’t the first period of interruption. It’s getting tiresome.

#2013 – Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

it’s been happening for the whole freaking week!!!!!! have u done anything at all??????? dont ever think that it’s a big deal for us to switch to another service provider, because we are!!!! its way more painful to deal with this everyday than just taking a couple hours to switch!!!!!!

My site is down again, for two days it was for an issue not related to the botnet attack and my support tickets were ignored. I eventually got Justin’s attention and he fixed it (our Rails gem was uninstalled). Now it’s something else– my site responds to ping, ssh and ftp but I can’t get it to come up in a browser. Is this related to the problems DH is having today?

After many years of being your client finally i give up on your network failures.

I cannot apologize so many times to my clients that your servers are down. sometimes emails sometimes websites..
you lost a client today. I’ll have to spend a while to migrate, but for sure will be planned time and not emergency random time.

Shame on you.

Please tell me I’m misreading this and you’re not only single homed, but peering with Comcast?!?!?

7922 7922 7922 26347
AS-path translation: { COMCAST-IBONE COMCAST-IBONE COMCAST-IBONE DREAMHOST-AS }
edge3.LosAngeles1 (metric 20040)
Origin IGP, metric 0, localpref 100, valid, internal, best
Community: North_America Lclprf_100 Level3_Customer United_States Los_Angeles EU_Suppress_to_Peers Suppress_to_Peers Suppress_to_AS174 Suppress_to_AS209 Suppress_to_AS701 Suppress_to_AS1239 Suppress_to_AS1280 Suppress_to_AS1299 Suppress_to_AS1668 Suppress_to_AS2497 Suppress_to_AS2828 Suppress_to_AS2914 Suppress_to_AS3257 Suppress_to_AS3320 Suppress_to_AS3549 Suppress_to_AS3561 Suppress_to_AS3786 Suppress_to_AS4637 Suppress_to_AS5511 Suppress_to_AS6453 Suppress_to_AS6461 Suppress_to_AS6762 Suppress_to_AS7018 Suppress_to_AS7132
Originator: edge3.LosAngeles1

Why isn’t your edge router responding?

Past 3 hosts have had similar problems. Left InMotionHosting.com after 5 years in January. Now on BlueHost–email blacklisted. Just setup a VPS account on DH last night but thank goodness I didn’t change the DNS yet. It’s very frustrating. Customer service doesn’t appear to mean anything anymore. Why can’t these companies figure out how to prevent these kinds of situations. Is it a lack of knowledgeable staff? Companies don’t want to hire the best and brightest anymore.

We are still having MySQL issues (since very early this morning) across all our sites

Dan

I See Cogent 174 and 10912 (no idea who they are) in paths as well

So what is the actual problem here? What’s the cause of the trouble? Why is it that we don’t know what’s going on for hours when these things happen? If these are directed attacks, why is it that Dreamhost takes so long to address them? Answers instead of “we’re seeing things working again” responses are crucial to keeping clients at this point.

Here’s the deal, Dreamhost: I don’t care what’s going on with your datacenter. I shouldn’t even have to know what datacenter my stuff is on. That’s what I pay you for. If you were smart, you’d have us all at several different datacenters and you’d have DNS failover ready when one of your datacenters fails. As it stands, you’re ruining a brand you built up really successfully over a number of years… and you’re asking the rest of us to continue to trust and come with you. Your support staff keeps apologizing, but it’s pretty hollow to keep apologizing, and then day after day for months on end still continue to let our sites go down for hours at a time. We’re paying you money. Yeah, maybe we’re not paying as much as we could at another host, and perhaps we’re getting what we pay for… but you have a lot of nerve charging us at all, ever, to have days and days of downtime on end. My faith in you was strong, and I cut you a lot of slack. A *lot* of slack. But these are now no longer isolated issues. Entire datacenters going down for days at a time? That’s a pattern… and it means there’s more wrong than just the technical issues themselves. This is a failure of leadership and planning within the company, and until there’re some more concrete solutions to that problem, you’re going to continue to hemorrhage customers like me that really, really were rooting for you. What a shame.

NOTE TO ALL USERS:

WE RUN A HUGE MONEY SITE AND MADE THE MISTAKE OF OVER A YEAR AGO TO SWITCH DUE TO THIS EXACT SORT OF PROBLEM. WE HAVE SINCE BEEN WORKING EXTREMELY HARD TO BUILD EVERYTHING ONTO A NEW PLATFORM AND ARE MOVING IN A WEEK OR TWO, FINALLY.

DONT MAKE THE MISTAKE OF THINKING THIS WILL GET BETTER; IT WONT. TAKE STEPS NOW TO MOVE, IT WILL SAVE YOUR CO. MUCH TIME AND MONEY. SERIOUSLY, DO NOT MAKE THE MISTAKE OF STAYING HERE.

DEFINITION OF INSANITY: DOING THE SAME THING REPEATEDLY EXPECTING A DIFFERENT RESULT.

I am on the east coast and am having problems with POP servers…. no I can’t even get to my DH login page…
wtf…

Anthony,

I wish you luck in your endeavours, and hope that you new provider will not get DDoS Attacks the day after you move

@Dan: We have multiple providers in each datacenter. http://wiki.dreamhost.com/Network_Infrastructure

Can you please tell us what caused the issue?

“Some of our customer in our US-West Data Center (Los Angeles, CA) may be having some connectivity issues at the moment. This includes web, mysql services, and specifically the homiemail-sub5 mail services (You can find this in the panel via Account Status on the top right, and checking “Your Email Cluster”).”

Are you serious? You’re not even reporting what is happening… I have been very supportive of DH so far…but this is getting crazy.

Our website is still down, and we have people on a web conference right now who need things from it. Not pleased: http://www.preventconnect.org

my sites are still down!

Justin@DH

What I see right now does not seem to tally with the Wiki, but that is another story – I would be good to know what went on

After several (we all know I’m being generous), of these types of outages, I started to dig around for competitive offers with other hosting companies. I am using a dedicated box here, and paying a high price for the hosting, so I expect better service, connection, and uptime.

For about $50 more, I was able to negotiate a deal with SoftLayer for a dedicated box. Let me tell you, SoftLayers support shits on DH support on any given day/time.

If you’re on shared hosting or paying <$20/mo, you don't have much room to complain. If you need different service, you can go to other crappy hosts, like HostGator, GoDaddy, FatCow, BlueHost, etc..

Anon,

Exactly, it makes me smile that these ‘000s, millions, and billions of $ a second companies (so it would seem), use DH as their only provider

All my customer’s sites down again? this is crazy…
where is hosted http://www.dreamhost.com ??? that works fine!

I start to be a bit tired of this network issues

Hi, Still having issues at 2:46 EST…?

I was hoping that a vps was more stable. But I was wrong

So what’s actually CAUSING these issues? You never touch on that, Justin/DH Team.

Several lost emails. Pitiful! =(

Susan

Your site seems to work for me

BOFH, realistically. DH was at such an unreliable state in the past few months, I got a redundant server set up to mirror the DH server. It was working out OK, as I would have cron jobs mirror the server files and the DB, but when the outage is during a cron job, or a backup process, my fail safe wasn’t so safe.

DH was good when they were at a smaller size. They made some big business decisions, and are now stuck dealing with major outages. If they were to tidy up their networking, and invest money into proper fallback solutions instead of trying to grow bigger and bigger, they would be much more efficient and favored.

I hear ya on the VPS inerba, that’s what I thought too. I have had better results with my clients on Dedicated boxes but only the ones that are in the US-East data center.

@Bryan: There is a meeting happening right now to discuss what happened and get details for the future maintenance to resolve it. As soon as I have more details to give out we’ll update the post.

Anon,

I agree that DH appear to have grown too fast, but as I have said previously on these forums, you have made redundancy preparations, most other people have not, and yet they claim that they are losing millions of $ per second. With that kind of statement, I honestly cannot take them seriously

I hope it will come out what was wrong, but unfortunately cannot off help due to conflict of interest issues

Hundreds of emails lost in the last 5 hours due to this mess. Are they lost forever or stuck in a queue somewhere?

Edd,

They should be queued by either the remote server, or the inbound gateways

@Edd: If it is incoming email, it will queue up on the senders email servers to try again later. If outgoing, it may be delayed due to connectivity issues.

The service has been very reliable for the last many months (following the imfamous period. :P) so I’m not too worried about it.

Thanks for the status update and I hope this glitch is temporary.

I’m back up as of about a minute ago.

May 15th, 2013 at 12:10 pmSimon Anderson Says:

Customers, I know these network issues the last two days are incredibly frustrating, because you rely on us to keep our services up 24/7/365. We’re doing a post-mortem right now to determine how to prevent today’s issue from re-occuring – we think it was a combination of line card hardware failures in and a potential bug or misconfig in some network software. But yet to be fully determined. We’ll be doing a detailed status post tomorrow once we know with confidence what happened and we ensure our fixes are stable and permanent. We’ve upgraded so much of our network and server hardware, software and systems in the last year that it’s equally frustrating for us to be in this situation where failures still occur. But we are going to beat this instability, and be better for you all. Sorry for my insensitivity tweeting about being at the WorldBlu event today – I was aware of what was happening with our network, and our best people and outside experts were on it.

BOFH, that other one is InterNAP. The looking glass I was using wasn’t showing the other peers, so thanks for that update Justin.

still waiting on e mails from this morning. so frustrating as we lost 9-3 all ready of our work day

“You may have noticed that we do things a little differently around here. There’s a reason for that.” lol :)

check this out here … http://dreamhost.com/about-us/the-dreamhost-team/

All other sites are down but dreamhost.com is up.

Post mortem = after death

Gee, hope you had fun at your event, Simon! Your company has made my day a living hell.

Buh-bye, Dreamhost!!

Downtime once, even extensive, resolved and fixed – acceptable and tolerable.

Downtime repeatedly with no transparency, repeated assurances that all is fixed right before downtime occurs again.
That’s just poor engineering.

another 30 mins and still missing 5 hours of e mails.

Well, this is it. We’re moving on. It’s ben an interesting decade guys, but it’s clear you’re dying.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Renewed external attacks affecting customers across all datacenters (RESOLVED)
10:45 am

Renewed external attacks affecting customers across all datacenters (RESOLVED)

Posted (May 14th, 2013 at 10:45 am PST) by danwy

We are seeing a new wave of attacks, similar to the ones that were causing issues yesterday. They have restarted and are causing new problems again today, and despite targeting wordpress, affect all customer on our web hosting services. Our admin team is aware of this renewed series of attacks and are already working on trying to restore services to normal. Again, we apologize for the inconvenience this causes you and we will post updates as soon as more information becomes available.

Update May 14th 4:33pm PDT:The update that our admin team pushed out has helped overall stability across our network considerably. As of now, the fix is running on all shared hosts and we are seeing a very positive result. We’ll continue to monitor the health of our machines, but the night and day difference hints that our team has very likely correctly identified and fixed the root cause of the issue. The attacks against WordPress haven’t particularly quelled, but they are not having nearly the negative effect on the machines they were prior to our latest patch rollout.

Update May 14th 3:04pm PDT:All shared servers have received the aforementioned fix and the number of alerting services have dropped significantly. We’re still monitoring the situation very closely, however, we’re very happy with the results of this latest update.

Update May 14th 1:30pm PDT: We’re still in the process of rolling out the fix to all of our shared servers. Early testing shows some promising results, but we are also preparing backup plans to implement in the event that there are still issues. More info to come shortly.

Update May 14th 1:00pm PDT: We are nearly finished with prepping a fix for this issue now and will be rolling it out to all of our servers in the next few minutes. We’re not sure how long this will take, but we will update this post soon with more information.

Update May 14th 12:30pm PDT: We have confirmed that one of the fixes we have prepared is working in our testing environment. We are in the process of prepping it now for rollout across our system. We’ll post more information regarding this as soon as we have it.

Update May 14th 11:50am PDT: Our admin team has determined that the best course of action at this time is to temporarily block all access to WordPress logins across our system until a more sustainable fix can be rolled out. Unfortunately, this means you will not be able to login to your WordPress sites for the time being (you will receive a 503 error). We are already in the process of implementing the fix and rolling it out everywhere and will re-enable WordPress logins as soon as we possibly can. We’re sorry for the inconvenience this causes you, but it is necessary for us to better get things under control.

Update May 14th 11:15am PDT: We have just pushed out new mod_security rules to help mitigate the effects of this attack. This should mitigate some of the attack traffic we’re seeing. Some of you are noticing 503 errors when logging into wordpress, and we’re investigating that at this time.

This entry was posted on Tuesday, May 14th, 2013 at 10:45 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

169 Responses to “Renewed external attacks affecting customers across all datacenters (RESOLVED)”

what’s going on? everyday the same problem…

Guys… can you at least tell me whether or not I can expect connectivity within a few hours? I think at this point, some amount of a refund is due.

Back here to complain. Because that’s what I like to do, and because that’s what I do best!

Oh! And because I can’t get any work done without my site being up. I had a big radio interview for the site’s launch yesterday, and a couple of articles are going to be published about the site today, too. The site’s been down most of the day yesterday, and a lot of today so far.

Every single day it’s the same issue… And considering that it hasn’t been getting better, the attacks have become more frequent and the damage is lasting longer and longer it just might be the perfect time to jump the DH ship.

Yup. None of my sites are responding, and I’m unable to connect to my SFTP either.

Thanks.

May 14th, 2013 at 10:53 amPGP_Protector Says:

I use a nice little app on my phone to monitor my site being up & down, the poor little app has been screaming for the last few days, with the sites going UP & Down like a yo-yo.

Welp, hopefully you guys are like me and have a lot of hobbies. Sadly, hobbies don’t pay the bills.

Hang in there, DH staff. Been there before.

Frustrating as yesterday was the first downtime I’ve experienced in a long time… And I think the first time that all my sites went down….
Then they’re back live, we blast out the “all clear” and boom… Down again.
Hopefully the restore is quicker today…

It’s been 8 years. See ya.

None of my sites are responding, neither ftp.

@PGP_Protector wht is the app called?

dreamhost? more like downhost.

I am not mad at dream host for this because it not their fault but I am extremely mad at hackers! Guys keep up the good work.

Yea this is starting to get very annoying, I hope I’m not losing business over this. Not that Ill ever know while my websites are down.

Not at all happy. Can’t even fulfill my client’s retainer adequately. I wouldn’t be surprised if I lose that client because of this. My other clients are screaming. You guys are dropping the ball big time over there. Sad to be on this end, witnessing your demise. Meanwhile, I’m risking one very large client and several others. Not the first time in a long time for me @Joe Scully. This has been a consistent problem for the last few months.

Umm, My three sites haven’t be affected. Can access them just fine on desktop and mobile devices. I have, however, made sure I installed the WP login security plugin a while back. So maybe DH isn’t totally to blame. There’s WordPress and such that have weaknesses that these codeKiddies are exploiting.

Everyday we are facing the same problem.. Please let me know if can resolve else we will shift the hosting

No problems here, perhaps because I don’t use WormPress?

@Mike Rhodes

You don’t have a clue

Hack me once .. shame on you … hack me twice shame on me …

Dreamhost should have this issue under control at this point. Most hosts got hit with the WP Botnet Flood a month ago and experienced similar downtime. But then they got things resolved.

Seems like Dreamhost has been wrestling with this for a couple of weeks now. I really hope they stabilize things soon because Dreamhost is great when it’s working!!!

Considering DreamHost is being ATTACKED, I hardly think anyone should be bitching and moaning. Yes, it’s a big problem for us. However, DreamHost has to not only FIX what’s been done but also PREVENT future ATTACKS. They are the victims here, not us!

Whining again. Losing milions of dollars, hundreds of clients, etc … from guys that left yestarday. Guys everything will be OK. DH-team c’mon guys. Make this go away.

Tuesday, May 14, 2013 2:11 pm. I have a totally static site and it’s down. Morons suck, but dreamhost has to have some way to deal with them. It’s like cockroaches; you don’t just close the kitchen.

@Tim

You’re clueless. A crucial part of web hosting infrastructure is being able to mitigate such attacks. Not to mention the fact that this is supposedly happening on every single one of their datacenters indiscriminately, and is causing downtime due to resource exhaustion at each of them? Nope. No way.

Go away.

My main WP site is slow, but functioning in a “will it load this time” kind of way. I installed a login security plugin sometime last week and have had my WP registrations turned off for spam reasons for awhile. It may be while it’s handling these attacks as it is, but even still, it’s going VERY VERY slow.

This has gotta be some kinda record. I’ll wait patiently. :)

@JF: That article does point out some of the increases that all web hosts are seeing. It is quite large in scale.

@Robby: Sorry, your server, pristina, is one of those affected by the higher loads. We have our entire admin team working on getting services restored.

Honestly people, most of us are paying $9-$10 a month. If you want the resources to keep your site up during a massive attack or if your web site is SO important to you or your business that it should never be offline, then you should manage your own virtual or dedicated server. You *CAN* get 100% uptime, but you have to PAY for it and it costs more than $10/month.

DreamHost cannot control who attacks their sites, the best they can do is respond to the attacks. Given their (DreamHost’s) size, they are a ripe target for this type of thing.

Be patient and remember this is a first-world problem.

@Dave: Thanks for the feedback. While the shared hosting is less expensive, we do want to provide as good of a product as we can, and we are working to improve services constantly.

I feel bad for whoever’s job it is to fix this, because they are no doubt facing some serious heat and pressure. Glad it’s not me.

That said, why in the HELL haven’t I received a single email about this problem? Your paying customers shouldn’t have to search through your website to figure out why their site is magically no longer working. This is what you call public relations. You acknowledge the issue, apologize for it, and explain everything you’re doing to fix it with (hopefully) an ETA on a fix. And you send that in an email.

Also, it would be wise to offer some SERIOUS incentive to keep your customers around, because it sounds like a lot of us will be jumping ship. At the very least, I think a partial refund of our monthly bill is due here.

Everyday we are facing the same problem. All websites are down. :(

I have 2 VPS servers, and all my sites are down. Most sites are WordPress, but there are some static HTML sites too. I rebooted both servers, but nothing loads. However, I can access FTP as well as webmail. It’s just that the sites won’t load in a browser.
Is this a separate issue, or are my VPS servers affected by the attacks?

This is unacceptable.

@Cassie: Customers who have submitted a ticket should have received a reply within the last 5 minutes, updating them on the situation, and will receive more email updates throughout the outage. We cannot bulk email our entire customer base though, as that many emails takes longer to send out than the outage lasts. We are open to feedback though and we appreciate it.

@Pavol: Can you give me a domain to look up your account?

You know what really sucks, having an actual website issue right now and not getting any reply from customer support because they’re making blanket assumptions that everything has to do with this attack.

Sure, one VPS holds only 1 site (BTW, I just set it up, at the time the attack was going on). The domain on that one is EctomorphBodybuilder.com
the other VPS hosts all my other sites, one of them being HomerunDesign.com

@Erik: Can you give me a domain to look up your account? If it isn’t related I would be happy to have it looked into and not grouped with the other tickets.

@Pavol: I’ll take a look and reply to you via email.

I appreciate this site is about getting affected DH users back up generally, but is there any specific advise you need to give WP users to reduce attack vulnerability. (As recommended by WP I changed all default admin user ID’s and pw’s last month. Is this more of the same for those that didn’t fix the problem last time or some new WP vulnerability?)

Thanks Justin.

Just fix it. I’m not going anywhere. Too much trouble to do so. But, fix it please.

@Ivica – You must not have any clients. :-) So while you criticize those who are seriously suffering through this (it’s not just DH) and you meanwhile fill your time with World of Warcraft and twiddle your thumbs because you would have nothing better to do anyway; I’ll explain why this is a pain and leaving isn’t as easy as just snapping your fingers: I have felt somewhat stuck here because I suggested this host a while ago and migrating is difficult. Price is not the issue. I have thousands of files and several hundred databases that would need to be moved. Some not even under my control anymore. About 100 other accounts that my clients own that I feel responsible for helping with in migration. I can appreciate that today’s issues are because of a WP attack, but when viewed along with all of the other outages over the past several months … I cannot overlook DH’s own faults.

The last few days have been really rough. Like others here, my monitoring services are working overtime, and my phone is pretty much constantly lit up with alerts. Best of luck fighting off the attackers, but I felt like I should at least submit a comment at this point. We only use Dreamhost for “non-critical” sites, but the level of downtime lately borders on making the service unusable. I can explain the problem to my end-users, but ultimately, a site that is inaccessible isn’t of much use.

I have been reading about the service outage this morning, and our site appears to be included in the issue. I have not submitted a ticket because I’ve been checking the status/updates and I figured service will be back when it’s back. But, should I submit a ticket? Do I need to?

Guys, stop blaming the wrong people. It’s the attackers who is/are to blame, not Dreamhost.

Thanks Justin, the one VPS is up and running. As you said, the other one may still be propagating. With all the downtime going on, I just was not sure what was going on. Bad timing, I guess. Thanks for quick assistance!

@Jennifer: If you wish to receive email updates on the progress of resolving this outage, go ahead and submit a ticket. If you don’t need any extra information than what is provided here, you don’t need to.

@Pavol: You’re welcome!

As the owner/operator of an ISP for 15 years, I can assure you that downtime hurts some people more than others, and some people, who rely on their sites for doing business, the damage is often irreparable. I have been happy with DH, until now. But if this keeps up, providing good service to my customers and the impact on my business will outweigh any previous good will and loyalty.

In other words, get it together, and do it now. If, as I did a few days ago, report downtime to DH and subsequently receive another email telling me how to control attacks on my domains, I’m outta here. That’s what I pay you for.

My website is still having issues…

@Cassie

They don’t have time to send e-mails. There public relations team is probably working to help fix this too. just follow @dhstatus on twitter and turn notification’s on.

No complaints today, running around .030 on html and doing lots of mysql conueries within php and cakephp. Lightening fast today. Rendered in .175 seconds on a large noSql php site. Perl & MySql is responsive too.

@Blair: Your webserver appears to be heavily affected by this. I’ll see what I can do to speed up the fix.

I want to keep dream host, I really do! So I think I will have two hosts just in case the other goes down.

Site is offline. We are not a wordpress site either. I made our site using straight html code, so why is mine offline if you have only suspended wordpress?

@Keith: The wordpress attacks cause higher webserver loads and memory usage, and slows them down in general. Sadly, even non-wordpress sites may be affected. I’ll see if I can help your site at all, but it may need to wait until the admins fix the overall issue.

I’m on gilmer and I have no 503 error, I’m working normally on my WP which is up with no problems.

Question: Does chaning your wordpress url (with a plugin) protect you against these types attacks?

Frustrating as it might be … it is not just Dreamhost … go elsewhere and you will find the same story …

http://www.bootstrapbusinesssolutions.com/blog/wordpress-websites-everywhere-are-under-attack/#.UZKJbklLs6I

@Peter: Changing the login url? The attacks generally just hit wp-login.php so if it prevents that from working, it has a chance.

Thanks @justin will try it and see what happens. WP Backend is OH SO SLOW, not a good sign

@Peter: No problem. Likely your site isn’t the only one (or may not even be one at all) being attacked, so adding it may not greatly affect the server load. The effort is appreciated though for the long-run.

@Justin, what impact should the last update have on FTP or static pages? My sites on Pistons have been having some issues.

Thank you dream host. Thankfully people can at least access my sites. I would rather people be able to access the site first. We can work on the log in issues later.

This is getting pretty bad guys… I don’t think we’ve had a single month where service has been uninterrupted. Will we be receiving uptime credit for this?

Also, why aren’t we being updated via email about serious attacks like this?

Well… do it. Block wordpress logins.

Load average nearly 1200 again on towns. On an 8 core box. Ouch. The website is crawling.

@Keith: Your webserver is getting a reboot right now.

Is this the reason my email distribution list isn’t working today? I sent emails out hours ago and they’re still in the queue. :(

My sites that don’t run on WordPress are not loading. Any word on how long it will take to fix this? At the moment only two of my sites are loading, and they are the two sites I have running on a different server. All of my other sites are on louisa, which has been a problematic server for me (lots of down time, sites load slowly or not at all, etc) since I was moved there in January.

@Victor: When the outage is resolved, you can submit a ticket for credit. If you wish to receive updates via email please submit a ticket. We cannot bulk email all customers updates about an outage like this, as that is a massive amount of email that takes hours to send, much of which will be flagged as spam and cause spam block issues, etc. We are emailing the customers who have submitted tickets with updates instead at this time.

When are we expecting our sites to come back up?

Server GRADY is completely down. Please fix! Nowhere does it say that servers are being shut down due to the attack!

My domain is not working very well, we cant receive any email. Is it working ?

Thanx

I appreciate all the effort that’s going into this. I realize that this seems to be a massive attack – none of the wordpress sites that I host/manage are accessible, not just the ones dreamhost.

But I agree with previous comments – why aren’t we being notified about big attacks like this?

May 14th, 2013 at 12:25 pmIsaac Zepeda Says:

Again? man you are becoming the worst hosting ever, I love the shell access and install things but this is going to far, I have customers that complain a lot and I cannot do anything, the only thing I can do is change my hosting provider. Get your shit together and give us a better service, this is bad for your business dont you see?

I know where the name Dreamhost came: works only in your dreams!

You are a disastrous hosting company colleagues laughs at me and clients scream. Not sure how others compare but I’ve never heard such horror stories and CONSTANT downtime.

Your service is a disgrace to profession. I paid in front for two years for what? Constant downtime and problems?

Now I really had it. I’ll give you worst publicity you can get and look that not a single person that I know host there anymore.

The good side is that I know you do not care.

For the past several months, service has been spotty at best. Our site is constantly down, and then we can access it for a few minutes…just long enough to start working on something and then OOOPS!!! DOWN AGAIN! If it was once in a while, we could probably deal with it, but this is a daily occurrence.
I am sure this is as troubling and frustrating for dh as it is for us clients, but there comes a point when service issues are simply inexcusable, regardless of whose “fault” it is. We are hurtling towards that point at breakneck speed right now.

@DH, keep up the great work guys. In 8 years, you have kept our various websites running smoothly with rarely any serious downtime issues. Patience from your customers and creative problem solving is all that is needed. Customers, let us all demonstrate patience with DH. This is an external world-wide server attack and not something DH can control. We are all in this together. I understand it’s frustrating…they are on top of it and doing the best they can. I for one thank you. I have much gratitude for your commitment to getting us all back online as soon as you can.

Crazy idea… Would it ever be feasible and (even a good idea) for WordPress to have an API that allows web hosts to keep their users’ WordPress system up-to-date? Just as an option, I mean. Mandatory updating would break some WordPress plug-ins, but stable releases being pushed and/or reminded towards Dreamhost’s customers could go a long way, don’t you think?

DownHost and WormPress LoLP

I’m responsible for upwards of 20 client WP sites running @ Dreamhost, and my monitoring service (UpTime Robot, which is awesome by the way) sends me an SMS when it detects a problem with any of them.

Thank got for my unlimited text plan, because it has been non-stop SMS messages since early yesterday!

@Tyler: The one-click system does this, if you tell it to auto upgrade it will do that for you shortly after new releases come out. It can break plugins and such though as well, so sometimes auto updating isn’t desired. There are some alerts/tools in the works for the future to help notify of outdated software and maybe go further in the future.

@Tyler: If I remember correctly, this is an option when you do the original 1-Click Install.

And while older versions of WP are presumably more vulnerable to routine hackage, even 3.5.1 (latest stable release) is affected by this mess.

Holy fucking shit – we are in the middle of an email send on the eve of giveBIG (a giant charity fundraising event in Seattle that occurs tomorrow) and this crap is happening again?

Yes, this is starting to cost us money. And yes, my Executive Director is going to demand that we switch webhosting if we miss out on this opportunity due to repeated DH tech failures.

Now even more bad news! I have Avast installed… it shows “catching” VBS:Malware-Gen. It appears to have infected my laptop when I went to my site (which appeared to be up until a few pages it hung). From that point on when I bring up about any website on either SlimBrowser or Internet Explorer… I get the MALWARE BLOCKED message from Avast. Although Avast says it’s been “moved to chest” and that the threat was detected and blocked… it is showing up on every computer on my network. I looked up the description of VB:Malware-Gen and it basicly said it will infect every computer on a network… and they indicated the only way to get rid of it is to restore to previous versions… but not sure. So… it appears that anyone that goes to my website will get infected. THAT IS NOT GOOD FOR BUSINESS… well, it occured to me that maybe soon I won’t have a business. I DO HOPE I AM OVER-REACTING!

Anyone else experiencing this VB:Malware-Gen attack?

@DivTec: Sorry to hear that. I believe this issue is specific to your site, there are no known widespread virus/malware/etc issues at this time, just the incoming wordpress login attacks. If you want to submit a ticket with a subject like “hacked site scan”, our abuse/security team will do a scan and notify you of any outdated software, suspicious files, etc.

Ok… I will submit a ticket. I sure hope it’s some kind of false alarm. I’ve been with DH for years… really never had a virus show up on my website that is on your server before.

This is the second time we have had issues with DreamHost and are now seriously considering switching services. The first time this happened, our website was hacked. Cherry on top, a malicious script that redirected all visitors to a pornographic website. Too many problems to justify giving our business to DH.

This is pants.

I moved from Posterous (and they were crap too) and now this……!

The most exciting part of my year and I can’t blog about it.

Depressed muchly.

Are you lot just sitting on your arses, coz it sure feels like it?

http://i42.photobucket.com/albums/e302/Frances144/969634_10151692489687265_724808800_n_zps42773c14.jpg

I just tried to logon to submit a “hackede site scan” repair ticket… but can’t log on. URL https://panel.dreamhost.com/index.cgi just hangs… nothing displayed. Happened yesturday too. Is it me… DH please advise.

At least DH is being upfront about it. Most webhosts would rather sweep something this sophisticated and complex under the rug and take the following customer migration and hit because they think they can.

The truth is, this is not some hack committed by your next door neighbor’s kid in his/her basement doing this for giggles. This feels orchestrated, tenacious and well-executed. These cyber-attackers clearly don’t care how much collateral damage occurs, just that a lot of it would/will happen…for all we know they only were trying to look for something specific?

Won’t be surprised if GoDaddy is behind the attack. Also start wandering if they sent all their incompetent employees to DH..

This is VERY sad. I do not know what happened to my beloved, best hosting service on earth. We have been customers 10+ years on your host and proudly recommend your services to our 20K+ members on every mail we send on our mailing list.

This is VERY tragic.

Man, I feel for y’all. Both customers and Dreamhost folks. It’s not DH’s fault they’re getting attacked, but it’s horrible for their customers all the same.

I’m an IT guy myself who watches this stuff happen all the time. Security for a big DDoS like this is always reactive – there’s only so much you can do to prevent it, because those malicious arsehats keep finding new ways to do it.

I just hope I remembered to update all my WP sites before this all happened.

I read that HostGator, GoDaddy, and RackSpace are all getting hit today, too.

@DivTec: Doesn’t appear to be having any issues for me. That’s odd. Maybe try from another computer in case you do have a virus.

@Frances: We definitely aren’t sitting on our butts, we have the entire admin, network engineering, datacenter, support and other departments all on and working to get all these issues resolved. Our apologies for the delay and related issues.

plz resolve this issues ASAP.

@JustinDH we are all with you, we know you are doing all your best.

Looks like I’m back up and running. :-) Thanks!

I know who’s causing the attack. He’s attacking the ip of the site that DH is hosting. He doesn’t want that site hosted on DH. These attacks will stop if you guys stop hosting a site on your network. Email the email that is provided.

I noticed it very much yesterday…. WHY didn’t anyone post this YESTERDAY. And yes TODAY too.

I understand if it’s out of your control to fix. fine. you’re getting attacked.

BUT IMMEDIATELY POST HERE WHEN YOU ARE! Yesterday! you know?! 5 / *13* / 2013!

Starting to see some improvements …

What happened to the post from yesterday where pretty much the same happened? Removed it because that would look bad? That actually looks worse.

The situation is frustrating, but you guys at Dreamhost support have always been fantastic and I don’t doubt that you’re giving 100% to resolving this once and for all. I admit I’m crabby like everyone else, but as a loyal customer since 1999 I appreciate your high level of service and continual improvements. I hope you’re able to put the kibosh on this nonsense once and for all!

With the nature of this problem, why would anyone switch hosts? This is frustrating in the EXTREME but imagine the frustration of going through the switching process and it’s happening there too. It looks like DH has somebody on here posting today. I don’t think that was the case yesterday (maybe wrong) but that’s a good idea. I truly do not believe that the other host companies are sitting around watching their servers running smooth as silk.

see it was just a matter of time to get it back to working. Good work guys.

In peroration of this happening again, I am moving my word press over the a separate sub domain (on dream host) and giving the main domain a html redirect to the sub domain so I can redirect to any website or pull up a were down page if neeeded.

@Matt: There is a status post below this one for yesterdays related outage.
@ACJ: It is there, there is just a lot of maintenances that are “sticky”‘ed to the top. See:

http://www.dreamhoststatus.com/2013/05/13/networking-issues-affecting-us-west-data-center-los-angeles-ca/

Dear Lord, if you existed, I would thank you for blessing us with dreamhost, dreamhost outages and the ability to post comment on the dreamhost status blog.
I am not allowed out of the house nor allowed to watch TV, so the dreamhost status blog is the only entertainment I have.

Also thank you and dreamhost for hosting websites that, when working, each seem to somehow bring in the GDP of a small country and yet are fragile enough that:
1 – they cannot be hosted somewhere more expense, despite the massive income loss during even the briefest outage
and
2 – the owners* would all rather bitch than switch !

[*} outgoing and vocal internet money mogul running their wordpress sites.

Some of you guys need to stop complaining. DreamHost is a great host and is much better than a lot of the competition. I am a web designer and have worked with clients who use all different kinds of hosting companies and let me tell you that DreamHost and Host Gator always come out as number one in shared hosting.

May 14th, 2013 at 1:40 pmThiago Colebrusco Says:

Isn’t anything you could do to prevent this?

My customers are claiming desperately since yesterday!

Thanks.

ARG – Extremely bad timing. In the middle of a big PR push and site is down. Not good. Not good.

I’m glad you guys are on the job.. Will you send out an all clear message?

I see now. Sticky posts about maintenance skew how a user would use this site. Such as when there’s problems. Really who comes here for maintenance updates.

@Chrissy Morin: Tickets submitted for the outage will get an all-clear message, yes. This post as well will be updated and resolved when that happen.

My clients really need to see their websites I host here. Any news on the Fulton server?

Ok… I like you again. But, for how long? Hopefully for years!

All my sites are still down. I am quite surprised at how often I’ve seen this happen in the last few months. Any update on when this will be fixed ?

Site is up a running FAST!

WE ALL GONNA DIIIE!

@ Julia
@ Ahsrey

Your webserver has been started and sites are back online. I tested the site that you provided in the blog comment. Please let us know if others are having problems.

My sites are DOWN!

My site is utterly down as well, getting a can not connect.
Could you please look into this as well?

What a bunch of overdramatic, self-centric whiners. I’ve been on DH for 12 years now and their customer service / attention to security issues like this is just as good as it’s ever been. If this OUTSIDE ATTACK is causing you inconvenience, sorry, but I am sure DH is doing everything they can to kill it – they DO know what they are doing and they want your business like almost no company I’ve ever seen, no matter HOW much of an overdramatic, self-centric whiner you are.

If your WP install was infiltrated and now everyone visiting your site is catching malware or something, well, from what I’ve read of this particular attack, it’s your fault for using an absolutely horrible admin password and/or the default admin username.

And if you are running your entire livelihood and a massive client base on a shoestring shared hosting plan, there are absolutely no words.

Well, maybe “hubris.” And a few others that are not very polite.

Keep up the great work DH, and godspeed with this latest crop of idiots (both attackers AND customers).

@TomsCigars

Your site is alive and well ;)

When do we know if all sites should be up again? Mine still not coming up.

the 2:10pm says everything is back. but all my NON wordpress are still completely dead.

My site is down too. It came back up for about thirty minutes and now it has been down for about 45 minutes.

Please try HARDER. Thank you.

Please try HARDER. Thanks.

Back up! good job

Tuesday, May 14, 2013 5:22 pm. My site’s back; I am so happy.

Now can’t you chase down these creatures and punish them?

Writing to say THANK YOU :)

Fabulous, thanks for the quick fix. I was only down about an hour. I’ve been with DH for almost 10 years (I believe) and you all have done a great job. Thank you for your prompt attention!

Up and running…

but… for how long? ¬¬

This is why I’m unable to connect to my charles-city ftp??

Website still down…

still down. any indication from where the attack is coming?

Now, my whole site disapeared from ftp! No more data!

Cool DH!

@Arturo
@Jenn
@NLane
@David Babson
@John
@NAllen

Your sites are back online. Please give it another try.

Where is my site??????????

I run no WP or Joomla, only static html and Drupal 7. Everything is down. For a couple weeks I’ve noticed an escalating dictionary attack coming from IP addresses registered to Baidu (in Beijing and in Mountain View) but I’ve also been seeing more and more bot-generated multi-probes for every form of WP-admin login in general use, coming from IP addresses all over the planet. So the probing is going after more than just WP… and it’s happening to more hosts than just Dreamhost. Imagine what it might be like to have a WP site mounted on one of those GoDaddy servers that hold upwards of 20,000 sites per machine…

oh good – for a moment I thought I’d written some awful php that had taken down the entire server (seriously – it seemed to happen every time I updated some files!) glad it’s not me.. hope you guys manage to mitigate the problem soon.

My whole domain disapeared! I had to recreate folders and put an emergencial index.html there.

@Paulo

I tested various sites on your account and they all loaded fine. Please give it another try.

Try http://mundoaz.com. There is a warning page which I just created there.

mundoaz.com domain disapeared.

you’re happy with the results ??? are you kidding me???

My sites are still down as well. ProRec.com is one of them.

Ideas?

Brent

@Brent – Your site comes up for me.

I still can’t access my WordPress account.

Where is my domain mundoaz.com???? I had to recreate folder in ftp because there was nothing there!

still not able to access some of my wp sites

Thanks DH! I can’t imagine the stress you guys have been under today.

@JF

Could you provide a specific domain or submit a support request and provide more details ( ex. domains that are not working )?

@Merry

Your site is back online. Please give it another try.

What’s really surprising is the email I got in response to my server outage ticket:
“We were unable to verify any server-wide problem related to the outage you’re experiencing.”
Really? You couldn’t find ANYTHING wrong yesterday?

DH, please, put a backup in my domain mundoaz.com! There is nothing there anymore!

@Paulo

I don’t believe your problem is related to the outage. I sent data restoration instructions to the email address that is listed on your account.

Thanks Brian.

My domain was OK until some hours ago and suddenly it disapered, including folder mundoaz.com in ftp.

Im doing a restore.

Thanks, DH. My sites seem to be back up again.

Brians, how about this message:

Unfortunately we were unable to process your domain restore for
mundoaz.com as there were no available backups!

No backups? No one single backup? My domain simply disaperead???

@Paulo, I’m restoring your files manually right now. As Brian S mentioned, your issues are likely not related to the outage.

@paulo

Your site has a picture of Santa. Use this downtime to update your site and make it useful and stop bugging people who are clearly extremely busy.

This domain is on a shared host but is not running WordPress. It’s been out since yesterday and is still unavailable now. I sent a support request through the control panel yesterday and I haven’t gotten a response yet.

Pleasse advise. Thanks

Press F5 in main page and ad will change.

Getting this – Fatal error: Access denied for user ‘username’@’steelers.dreamhost.com’

6:26 PM (CST) still slow (very slow), although I can actually see the site (although it can take over a minute for a page to load). Hoping y’all are able to solve this one soon.

@Chan Do: I’m locating your supprot request now, please expect a response shortly.

@Paulo: Your files have completed restoring.

@Adam, I’ll look into that for you.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » External attacks affecting customers across all datacenters (RESOLVED)
11:19 am

External attacks affecting customers across all datacenters (RESOLVED)

Posted (May 13th, 2013 at 11:19 am PST) by Dan J

Some of our customer may be having some connectivity issues at the moment. This includes web and mysql services, but no mail services are affected. We are investigating the root cause and are working towards resolving it. Please accept our apologies for the inconvenience this is causing and rest assured that we will post updates whenever more information becomes available.

Update May 14th 10:38am PDT: We are seeing a new wave of these attacks that are causing issues. Please keep an eye on this new status post for updates.

Update May 14th 12:38am PDT: At this time we’re seeing almost everything back to normal. We will, however, continue to monitor everything to be sure and will not consider this resolved until then. Please contact us (Support -> Contact Support in the admin panel) if you are seeing any issues with your service.

Update May 13th 11:43pm PDT: The focus of the last few updates has been mainly WordPress, so I’d like to provide some context as to the nature of the issues at hand for those that haven’t read far enough back in our numerous updates:

While your site may not be using WordPress, you may still be affected by slowness or downtime. The attacks are targeted towards WordPress, but the nature of them caused many hosts to run out of available resources. Several servers are still in need of individual attention to make sure the fixes were applied. In the meantime, those hosts may be fluctuating so it’s possible that even non-WordPress domains are affected.

Update May 13th 11:05pm PDT: If you have been experiencing 503 errors while logging in to your WordPress Administration Panels, this should be resolved. Our blocklist was cached and we’ve updated it to a current version. If you are still getting 503 errors in your browser while trying to login to WordPress, please contact Support through your Web Panel.

As far as our overall progress on blocking the malicious WordPress traffic goes, we’re continuing to identify hosts that are not functioning properly and get them up and running with our implemented fixes. There has been great progress made on behalf of our stability team, but a few hosts continue to need a human eye and we’re taking them in turn. There are also services spread out among many other stable hosts that need attention, so we’re not quite 100% but the majority of our customer’s sites should be functioning properly. If your sites are down, please remain patient until this status post has been resolved. Thank you everyone for your continued support and patience.

Update May 13th 9:05pm PDT: We are making progress in giving machines individual attention as they require it. Many machines are still experiencing resource starvation issues, requiring us to manually push fixes as they are not accessible to our automated tools. The implemented fixes are making a huge difference in the stability of our network.

It has come to our attention that some customers are unable to access their WordPress administration panels due to the blocklists we have implemented. We are trying to determine if the blocks are over-zealous, or if we can request unblocks easily on an individual basis. Our administrators are currently looking into solutions that will lower the amount of customers being blocked.

Update May 13th 7:46pm PDT: We’re seeing improvement across most servers with a few still requiring manual intervention that we are working to resolve. We’ll provide another update once we have more information.

Update May 13th 6:35pm PDT: Our Systems Engineering team continues to roll out the security updates. So far so good, please stay tuned for more updates.

Update May 13th 5:51pm PDT: We have adjusted our security rules on incoming web connections to help mitigate WordPress attacks. These security updates are currently being rolled out across all servers and we’ll be monitoring the servers as the updates happen.

Update May 13th 5:22pm PDT: We are continuing to explore and implement a multi-faceted approach to the attacks we are receiving. Progress is happening and early results are very good. We’ll provide another update shortly.

Update May 13th 4:45pm PDT: We don’t have anything new to report at this time, but the admin team is still working on improving connectivity across our data centers. We truly appreciate your patience while we work through this.

Update May 13th 4:17pm PDT: The fix is helping and we’re seeing more improvement across servers, but the admin team is still working on a more aggressive solution on mitigating the brute force attacks. We will keep you all updated with any new information as soon as we can.

Update May 13th 3:50pm PDT: The admin team is looking at several short term options to mitigate the attacks even further, but also investigating long-term solutions. They’re working diligently on getting things back to normal and we’re seeing a a bit of improvement on several servers. We appreciate your patience with us and we’ll provide more updates shortly.

Update May 13th 3:18pm PDT: Our admin team is continuing to roll out the fix and monitoring where needed. They are confident the source of the connectivity issues are due to large-scale brute force attacks to wp-login pages. These attacks are overloading affected servers and the fix being applied will limit the rate these attacks are hitting wp-login pages. In addition to the fix that’s being applied everywhere, we’re also mitigating the attack by blocking IP addresses all around our data centers. We’ll update you all with any further updates as soon as we have them.

Update May 13th 2:55pm PDT: Our admin team is rolling out the fix and are monitoring servers as the fix is applied. We’re also manually applying the fix as needed and will continue on working on a more permanent solution. We’ll update you all with any new information shortly.

Update May 13th 2:20pm PDT: We’re still working on pushing out the fix and do not have any new updates at this time, but we’re slowly seeing some improvement with connectivity to several servers. We’ll post another update again and provide any new information that we may have.

Update May 13th 1:45pm PDT: Our admin team is still working on rolling out the fix mentioned below to the affected servers. We don’t have anything new to update at this time, but we will update this post again shortly with any new information that we have.

Update May 13th 1:20pm PDT: Unfortunately, we don’t have anything new to report since our last update. We are still in the process of rolling out the fix to all of the affected servers. The testing we’ve been doing has been promising and we are working on getting things back to normal ASAP. We thank you for your patience as we continue to work on bringing things back up.

Update May 13th 1:00pm PDT: We are sorry that this is taking a while to resolve – due to the nature of the fix it takes a bit of time to deploy in each case. We are of course rolling out the fix as quickly as possible and it does appear to be working on those servers to which it has been applied.

Update May 13th 12:40pm PDT: Testing continues as we are adjusting the number of processes per class handled by the Apache FCGI process manager (this in turn helps to prevent large amount of traffic tying up the apache service which results in over-utilization of CPU time and memory). The most recent reports from our system engineering team indicate that this is working but we’re going to confirm and then apply the fix across the board. We thank you again for your patience and will update you further as we make progress.

Update May 13th 12:15pm PDT: Testing of the solution continues and we will provide more updates as soon as possible – we do apologize for the continued disruption of service for those customers on machines which have been impacted.

Update May 13th 11:50am PDT: We are in the process of applying and testing a solution for the current issue. We’ll provide additional details once we’ve been able to gauge the efficacy of the fix. Thank you for your continued patience.

This entry was posted on Monday, May 13th, 2013 at 11:19 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

336 Responses to “External attacks affecting customers across all datacenters (RESOLVED)”

Guys, I’ve been a long-time fan, but it’s really to the point where it looks like your infrastructure is failing constantly… Like it’s held-together with bubblegum & bailing wire?

Agreed. DHStatus should *not* be showing up in my frequently visited sites list in Safari…

Interestingly, my sites appear to be unreachable and they are in the US East data center… maybe it’s not just US West?

Can I get my hosting moved to where dreamhoststatus.com is located?

My sites on webserver toombs are down

I have a site at Virginia data center that’s also down.

Updating the problem, my MySQL server is Down: peppermill

We are now looking into all of our data centers, as it appears it isn’t the US West (Los Angeles) the only that’s affected. We will be updating the status post with new information soon. Sorry about the inconvenience!

Sites aren’t down, but we’re having slow connectivity times reported across the US by our clients. We’re on East Coast telfair.

Sites just went down.

Mine are down, too.

Actually, they just came back up.

sooo check that out, they have network maintenance on 11 and everything crashes on 13, nice

oh and @Dustin Dreamhoststatus.com is hosted on Linode.com. Take a hint!

Same scenario! It’s been going down every two weeks.

what the hell my all sites are down hell to DH

Cool i have about 200 domain down. my clients are so happy right now, my phone is melting.

http://poslarchive.com is unreachable (or perhaps barely reachable – if I keep trying, I can reload once every ten minutes), and I have a whole bunch of annoyed Scrabble players from around the world trying to follow what’s going on at the European Open Scrabble Championship.

Well, it’s important to us! :)

Was up for a minute… now DOWN AGAIN

This really sucks. I have more than 50 sites that are down right now and my customers are freaking out! Get it together guys…

This seems to be a weekly occurrence with my sites. It was solid for a few months and now every week there is a window of down time and since all of my URLs are hosted on the same servers this can get a little frustrating.

This is ridiculous, in the last few months it’s been downtime every week for one reason or another. And half of those get middling support response. I had been staying with DH because of the ease of it, but this is hardly easy anymore, it’s becoming a miserable chore.

May 13th, 2013 at 11:50 amJuan Calderón Says:

Hi My sites with domains and bases in US – East (Ashburn, Virginia)are down too

In the last 2 months, 3 times down events with my personal site and now a trouble with all domains, I am very frustrated with the service right now

Seriously, it may be time to rethink your strategies on updates. You are killing me and I am losing clients because you guys can’t keep things straight. You really know how to kill a start up…

My site is down too.. no wonder the traffic drops since yesterday…

I agree with James. This is getting ridiculous. I have two clients in particular that I convinced to move to Dreamhost from GoDaddy and their sites have been down twice. Its making them question my credibility.

My sites have been going down constantly over the last 2 months. This service has become ridiculous and unreliable.

ITS GETTING WORSE … sites that were up are now all down.

so freaking annoying. it’s one thing that my personal site is down, but that all my clients’ sites are down as well? embarrassing.

EEK!! All sites are down. :(

It’s been the THIRD freaking times this same thing happened for the past week! totally unacceptable! and i was suggested by one your agents to move to another database server. WHAT???? To go through all the set-ups and let it go through the site for another 48 hours before the site is properly running! extremely frustrated!!!!!!

Is this causing VPS warning emails to be generated and server reboots when memory seems no higher than usual or even close to the limit at all ??!

Are you kidding me with this? Another outage. I’m building a website that is intended to be an e-commerce, multiple membership site that needs to have reliable uptime – how are we supposed to convince potential site members to sign up if this erratic uptime is what is to be expected from Dreamhost? This is absolutely unacceptable and a very poor showing to paying customers who have clearly chosen your services over places like Godaddy or Network Solutions. Is there any form of guarantee we can get from DH for uptime? Or should I just start shopping around for different hosting and just move my entire site off DH?

Unbelievable.

I’m leaving. After the outages last year I began diversifying my client base with other providers, but have kept Dreamhost for my primary. Since I have been with them for several years. But I’m pulling the plug. Downtime happens, but mass long-term blackouts is not something I can live with. The price is right, but the service is not. Sorry guys.

I’m also really upset! I’m paying basically nothing for perfect uninterrupted service that I’m reselling for mad profit. You’re going to force me to start paying for the service levels I expect as a business instead of choosing you, as the cheapest possible provider as I did now!

<– DH Customer since May 2006. May not be much longer. Tired of them having problems.

I’ve been with Dreamhost for years, but the last 18 months have been stressful and embarrassing. I moved one of my sites to another hosting provider about 2 months ago and my site is now amazingly fast. I am going to move all of my sites away from Dreamhost as soon as I can.

Add my voice to the chorus of web admins exasperated by the frequent outages. I’m taking my org’s websites off of Dreamhost. Any recommendations for alternative hosts that are WordPress-friendly?

I have two websites on the same server and the same user: One is available and one is not.

The one that is available is a simple HTML static site. The more complex site, which uses PHP but NOT mysql is not reachable.

I have one other website that does NOT use PHP and that website is available.

Anyone for a PHP fix here?

I’m not buying it. Dreamhost VPS pricing is not that far off what you pay for other VPS services. The difference is that they other guys don’t give me 3 days of intermittent service during a downtime. Bottom line: “We have 99.99% uptime…”, subtext: “…except when we are down, then you are totally up a creek”. When you purchase hosting, you are not paying for the uptime, you are paying for the lack of downtime. No matter what the price, it is a reasonable expectation. A hiccup here and there is normal, but these types of outages are a bit on the excessive side. So yes I’m upset, and if you are not, then you are just enabling them to continue selling unreliable service.

I don’t think I will be renewing next year.

How about a status update???

OMGOD I’M LOSING 10000000000$ A MINUTE FOR EACH MINUTE MY SITE IS DOWN. PLEASE FIX THIS ASAP.

My sites were back up for two minutes and then down again. Yes, please let me know of some alternatives. Between the slow speeds and the downtime things are unacceptable. I literally *cannot* work right now because all of my sites are down.

Answer and update please?

I have been a long time customer but I am going to have to start looking else where if this continues to be a on going issue. I can understand once ever you know year or something but recently you guys have been hit with a lot of down times. Is there going to be any kind of compensation to follow all this down time? I know my business can’t keep taking hits like this before I have to move on. I hope this gets resolved quickly as I am working on something currently but now I am unable to because of the outage.

Dale JR your making $10 billion per minute and still hosted at dreamhost?
Amazing

the bi-weekly sql error. dope.

Quit the mimimimi e start migrating your fucking account to a fucking new hosting!

I hope that this is resolved soon.

ALL sites down in Florida. Embarrassing

Thanks for updating us; I was confounded when I tried accessing several sites and WordPress said the database was down.

DH it’s been 30 minutes since the last status update. WHAT THE HELL IS GOING ON???!!!

Again? Few months ago I had problem with Irvine host.

Anyone any idea to migrate to a new host? GoDaddy? BlueHost?

ditto on all the comments about the crap hosting service this company has become…. the ONLY thing that has stopped me from moving to another host is the massive time suck it is to move 50+ cms based websites

I get so many downtime notices from my monitor service its a JOKE….. Im finally at the point that Im willing to deal with what a pain in the ass its going to be and move…since I cant get any work done….

I think Im going to do some host shopping!

Guys, I’m a huge DH fan but this is getting beyond ridiculous. Plus, there have been rolling issues with individual domains we have hosted with you guys and the support reps are like rolling dice…either a complete win or a loss. I even had two try to make me upgrade when it turned out (thanks to a great rep) that found out it was a DH fail…none offered a refund for 4+ days of down time.

May 13th, 2013 at 12:26 pmThomas Hanser Says:

It’s not enough that I’m unemployed, broke, in DEBT and can’t find a job, and now my site is down. Again. And I have interviews and people who are supposed to be looking at my site.

I’ve been with Dreamhost for ten years, but I can’t put up with this again.

These disruptions make me really sad, it was all working great for a few years now.

So what are some good alternatives to DH

May 13th, 2013 at 12:28 pmI can't believe... Says:

I can’t believe that it’s been more than an hour and no one has pulled the “NightmareHost” card.

The trolls are slipping.

My site has gone down four times in a month, but I think the worst part is that no one at Dreamhost will read, respond to, or care about this comment. I have been a customer since 2005 – over eight years – but I’d love recommendations for another host. This is beyond ridiculous. I just launched a brand new site at the beginning of this month, and simply cannot afford any more downtime.

again? seems like something happens every week and my sites go down. if it’s not a server failure then it’s a network failure. wtf?

May 13th, 2013 at 12:35 pmDreamhostSucks Says:

Geez. You suck, Dreamhost.
I used to send all my clients to you, and now I specifically advise NOT using you.

I’m switching to Media Temple.

Ah, so it’s not on my end… Thanks for the update – very handy to have the Twitter feed to keep an eye on highly incidental server issues. Well, stuff happens so I’ll just lean back and let the pros at DH deal with it. Life’s too short and I’m overall very happy with DH so I’m not sweating it.

funny that the dreamhost sites all seem to be working perfectly

I have decided to speed up my off load from DreamHost to kbg-one-hosting.net (very good one hour or less turn around on support) and kbg-total-host.net (system control very similar to godaddy).

Shame as I stayed with DreamHost for some sites as I don’t like to put all my eggs in one (hosting) basket

Also, when you guys are done fixing the outage, there are some really funny comments above that need a “Like” button. David O’Dell.

I have been receiving an error intermittently for about a month (usually in the evenings). I submitted a ticket and was told everything is ok. I’m sure it isn’t. Alternative hosting recommendations would be appreciated.

My site is down. Im in Canada. Regardless, in four hours Im heading to a Behance event and sorta need my site, by sorta I mean I absolutely do! I hope you guys have people around the clock, time differences aside, you need to be on this. It really hurts those of us that use our sites to run our businesses.

http://lifehacker.com/5911651/five-best-web-hosting-companies

I was just reading this on lifehacker…. the: “a small orange” company looks good…

they remind me a bit of this company a few years ago before it was all about growth, and they forgot about quality anything

My website is still down and customers started complaining… shamefull

Guys, I’ve been with Dreamhost since May, 2004. Rarely do I Have issues. Calm down and let DH deal with it. I’m sure the sites will be back up soon.

This is extremely poor timing – on the very day that we are publicly launching our new website. This needs to be resolved quickly.

I’m sorry everyone I know I’m responsible for the Murphy’s Law incident here. I NEVER should have done the Happy Dance when I realized the bosses were gone today and I could get a shit ton of work done. I jinxed us all!

Drat! Dreamhost…you’re making me look bad…

my link is a image gallery I used to run at DH , they shut me down for the little time they used to be online and it was slow. I used to get 12000 visitors daily. I have since moved to dedicated hosting but ofcourse (been there for 4 years now) I lost all traffic except for the ones who remained loyal.

move out of DH as soon as you can. If you have business or think your site is becoming a success, GET OUT

Site down, forums down… and this is NOT COOL. We are currently a community of some 20,000+ neighbors in a time critical issue involving local government negotiations. The failure of your services is a potential killer of this effort which, if it fails will result in the loss of millions and ruining the lives of nearly 50,000 residents. Can you PLEASE be a LITTLE reliable?

btw , move to linode. there is no reason to use dreamhost,bluehost, fart host when you can learn a bit and get your money’s worth. Linode is far cheaper if you look at it beyond the cash value.

This is the third time in the last two weeks go down and we host more than 50. This becoming a huge problem! I have used Dreamhost for the past 4 years and didn’t have many problems. In the past 6 months it has been a roller coaster. I hate having to keep explaing to my clients that we can’t do anything about their site being down!

Chris – we just set up a VPS with Known Host and are working on switching all of our sites. They are a little more expensive, but from what I read it’s worth it.

Pffff Dreamhost reculiao son como la mierda, me harté los respeté y esperé mucho tiempo pero ya son un pain in the ass. Me cambio.

If you at Dreamhost have the same equipment as when I first suscribeb to yours services, you then know the cause of the problems….
I too host some sites that are at a critical time, as http://gvturbulence.com, a vocal group who is only one week away from its show, and has a member’s section with all the videos and partitions for home practice…

Agree with Ali, this is happening too often, Im starting to think of get the hell outta here. I used Dhost for the past 6 years, and 2013 is the worst year by far. Please, explain and fix this asap…

Any updates? It’s now 4:00 on the east coast and we’ve been down all afternoon…

I’ve got several sites on DH and they’re all having problems.

Dreamhost, I don’t want to move my site but these down times are costing me money.

Looks like avengers is just completely dead. The SSH port never connects, and the server is not pingable.

Even control panel functions are hit or miss today. Good thing I have some html front ends to handle initial landing. WordPress requires dozens of sql transactions just to display a home page, and are more likely to fail in a rpc database environment. Still, I just ran through 30k MySql reads in 20 seconds with a php page, which looks good. Not sure a rough patch of a few hours is the end of the world, I think I’ll stick around.

May 13th, 2013 at 1:16 pmPatrick Houlihan Says:

The support quality has declined as fast as the reliability.

Just 4 days ago my forms stopped sending emails. The first response from support took over 24 hours, assured me it wasn’t because Dreamhost was experiencing email problems, and that I needed to make some changes to my forms.

Two days later, after I wasted a lot of time troubleshooting, the second DH rep blamed it on Dreamhost’s email outage.

Neither rep had the competence or friendliness of the reps I’ve dealt with in the past.

Sites back down now (It seemed like everything was fixed about 10 minutes ago ) .. now it’s all offline again.

All of you folks who keep saying “I’m leaving Dreamhost” blah blah, well, get up off your butts and do it! I still have a couple of “small” websites on DH still, so stuff like this no longer bothers me, but moving away was one of the BEST decisions I’ve made in a VERY long time.

Well-done, Dreamhost. I applied for a job today and sent them my website to look at my resume. They cannot reach it and can’t see my portfolio and they’ve cancelled the interview we had tentatively scheduled for tomorrow because they can’t see if I’m what they’re looking for. I think this downtime just cost me the job. Unacceptable.

What particularly bothers me about this is the ambiguity in the information provided. Seems to be politician in a economic crisis and not engineers and programmers.

Also there is another matter “behind” this: I’m using DreamHost since three months ago and is the third time this occurs. Not only is it unacceptable to have interruptions in services, with the usual consequence of loss of visits and money, but at the end the problem, always note “additional constraints” in regard to traffic, RAM and CPU assigned to shared servers, etc… Therefore, we must expect a worse experience each time that this happen.

That is, they try to hold more and more users with same equipment and not to invest to provide reasonably good service.

I’m starting to regret for having chosen this company as a provider of hosting for my websites.

People,

This is really getting difficult. I need a hosting server where I do not have to worry about infra, etc. etc. Nowadays you have FREQUENT problems and I have customers running on my hosting an they are complaining A LOT. I need to know what is going on and see is this is going better in the future. It will take me toooo much time to migrate all my hostings but I am sincerely thinking of taking the weekend with some friends to have that job done.

Another downtime in only a few days. I am already searching for alternatives on WHT.

LOL Dustin, u hit the nail on the head.

I am wondering if Dreamhost will be giving account credits for all these outages. I have 3 or 4 sites that are down right now, and people aren’t happy.

Anyone have recommendations for a different host? Post some links, please.

come on guys,

My podcast is launching this week and it would be a shame to not be able to launch it because of my server is down.

This is some of the most fun I have seen in years…

People making $10 Billion a minute…. Buy your own data centre….
People threatening to leave – If you want to leave… leave…

I have been with DH since 2006….

I work for a hosting company…..

For the $$$ I pay its a reasonable service…..

@William, there is a site called ‘Google’

2 HOURS … STILL DOWN …

– WHY WOULD DREAMHOST SUDDENLY HAVE TO ROLL OUT FIXES TO ALL SERVERS ONE AT A TIME???

– AND IF THAT IS HAPPENING WHY HAS MY WEBSITE BEEN UP AND DOWN ALL AFTERNOON?

How about providing some more detail on what is going on??? Like really??

@william the next best place I may be looking into is bluehost.com. there system status is a little more on the ball in terms of being up.

Hate to kick a guys/company when they’re down, but get it together! I could do without the extended outages. I don’t use my sites that much, but I wonder how frequently my site are REALLY down. I bet monitoring would tell me secrets that I’d rather not know. I loved your blog posts, I love your control panel and you all seem like a good bunch of guys. I hope you all pull it together before the company is in real bad shape.

P.S. I’m wooting for you guys!

-Herman

Just arrived from Hostgator, let me come back guys. See you soon down there

@John
I remember once upon a time, DH would give a free year of hosting for this kind of downtime. I wonder if they remember that… because I sure as hell do.

Tired with failures I moved my sites to two hosting accounts (which I got from Network Panda and GoDaddy) last week. I hope everythning goes fine and till now I am satisfied.

I only had one last site on DH and moving it now.

The kind of business I run on my sites does not tolerate such extended outages I had with Dreamhost. I was losing money and customers I couldn’t put up with this anymore.

No offense guys, but your service is becoming more and more of a drag. I’ve stayed with you guys because you were always stable enough at a fair price, but now I may need to take my business elsewhere.

@me I don’t want to leave, I love dream host, but this is annoying.

All, (with respect)

f you are running a business, (especially e-commerce) get a VPS with at LEAST 2 separate providers, and replicate your data, and use DNS to send your customers to alternative sites

Its common sense

How about some real info instead of this? These updates don’t actually say what the problem is, there’s a mention of Apache being the cause, but this is across a lot of servers at once, Apache runs independently from machine to machine, is there an attack on infrastructure or something? No idea to know. More info would be better than hollow apologies.

I’m not buying the explanation. What on earth does Apache have to do with VPS being unavailable by SSH or ping?

Seriously, it’s up for a minute, down for five.

This is the fourth time in a month that all of my sites have been down. It’s not good for my clients. I was told to upgrade to VPS. I don’t need to upgrade, you need to fix the issues. I never had these issues before the last 6 or so months. I’ve been with Dreamhost for YEARS! This is not looking good whatsoever.

@Cthulhu

If, as stated, it is taking the CPU and memory to 100% it is possible for ssh to fail (out of resources), and pings (ICMP) are usually the first to go when their are load issues

Why would Apache have anything to do with EVERYONES sites being down all at once?? Or my VPS going down??

I mean VPS Apaches independently per VPS doesn’t it??

I’m getting Network timeouts … sites are up for 10 minutes down for 5 … rinse and repeat.

May 13th, 2013 at 1:43 pmPedro Henrique Says:

Seems like a bitcoin is attacking all DH Data Centers.

May 13th, 2013 at 1:43 pmMonkey Phalice! Says:

Is there a company that can handle the migration to a new server for me? That really is the only thing stopping me leaving right now… LAZINESS!!!!!

DH has a 100% uptime guarantee! [REALLY???] I would really like some of my money back. I’m not renewing, that’s for sure.

“If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!”

I have been a Dreamhost customer for many years and I never thought I would consider moving elsewhere but this downtime is unacceptable. The service has been really unreliable over the past week and it has really affected my business. I am supposed to demo a site to a client but instead I am writing this comment. Completely unacceptable. Get it together DH or you have lost another customer.

40 minutes since last status update …

@Sukrit

Open a support ticket

Disappointing that the service levels are fluctuating so much recently. If you have a business running on DH, I highly suggest you move it elsewhere. Not to sound crude, but I am super happy I did that during the last outage. I still use DH for personal sites and prototypes but not for a service I need to be reliable and responsive to my support requests.

A longtime customer

Yup… our website has gone down the past 3 business days. The first excuse was a “DOS” on the one DC. (pshaw). Yet, our server load average was stupid high. The next day’s excuse was simply that our server was “a little busy”. Quoted from the support team’s response. A little busy, huh? Load average of 1200+ on an 8 core box is like 150 times how busy it should be. I’d call that a little busy. Now, I feel like you almost admitted what the issue has been all along. You guys have been tweaking the number of FCGI processes across the board, huh? Now this makes more sense.
Anyway, I’ve got a ton of people coming to me and saying we need to switch hosting providers. You guys can’t seem to keep a webserver running. So… yeah.
Guess I’ll start shopping around.

All of my clients are using Dreamhost and have been for the past 7 years. I’ve consistently backed up Dreamhost during these outages, but it seems to be happening every single week now and for longer and longer periods of time. At this point, I am starting to look like we are suggesting and using the CrackerJack of web hosts. It used to be a good service. I’m interested to know other hosts you guys have switched to with greater success. Price is not an option at this point. My reputation is more important. Sorry Dreamhost, but I can no longer stand up for you. You’ve failed, miserably.

Website is completely down, its not on and off, just off!!!!!
Any update on this? I haven’t see any update for 4 hours!!!

We’re sorry for the continued problems everybody. We understand this issue is frustrating and we are doing everything we can to get things back to normal and determine the root cause of this service disruption. We’ve made progress, but still have a ways to go. We’ll try and keep you as up to date as possible through our updates.

again….

ask for some credit to your account. last time this happened it caused data loss and they gave me credit.

COME ON GUYS FIX THE FCKN SHT!!! IS A BUSINESS!!!

how do you open a ticket if you can’t even get to the ticket system?

May 13th, 2013 at 1:57 pmMonkey Phalice! Says:

@Cristian I think your caps lock mAy Be StuCk?

Considering all the work in moving a site, is there any hosting company where this doesn’t happen? I mean they are all promising everything to everyone for almost nothing in price. That cannot be done. Having said that, DH does appear to be in serious trouble. Outages are not a surprise, I have come to expect them. The question now, where to switch. I do feel sorry for the people who work for DH…the company is obviously stretching their resources past what they can provide. Too bad.

hey dream host CEO

if you cant handle this business and cant make servers up and running then please quit, and give someone else chance to manage DH.

please my all websites are down , stop is stopped , all FTPs are also down, MySQLs are down.

don’t you care ???

I suspect either bad hardware or a DDOS. This goes way beyond what I’ve seen from servers with fully loaded CPUS. Connections stay stuck in SYN_SENT until the connection times out, which means I can’t get a single packet in and out over the course of a minute. Unless the Linux scheduler is a lot worse than it used to be, it shouldn’t be possible to get things into that state from a user space run amok.

I’ve been busy fending off chinese IPs from my forums for a week now. I wonder if that’s the large traffic they’re ambiguously talking about, heh!

@Bob

Admittedly they may be stretching their resources too far, but they have grown extremely quickly

I have moved from 4 ‘budget’ providers recently after continued DDoS attacks in the data center they were hosted in. No one is immune to the ‘Bad’ Internet, and script idiots out there….

5 hours later, and all my sites are still down. :-( I was just telling a friend the other day how problems never last more than 20 minutes. Now I’m not sure what to think. The loss in business, on a monday afternoon, is considerable.

I find it particularly disturbing to see an update like “We don’t have anything new to update at this time”. If you had a plan for handling this type of crises, then you should be able to report what you’re doing every 10 minutes with specific details and estimated resolution times. Actions speak louder than words so…you’ve just lost this 11+ year customer.

This seems to happen a bit too often for me. Besides the fact that it has been weeks now where it’s taken too long to connect to my websites (multiple) it really makes me want to find another hosting service. But frankly I purchased a really good plan for cheap so I guess I should expect this, huh?

#FAIL. I’m not renewing.

@Mark

15 Minutes or so ago they said they were rolling out a fix, do you want someone to post on a server by server basis ?

Me Says:

“Admittedly they may be stretching their resources too far, but they have grown extremely quickly

I have moved from 4 ‘budget’ providers recently after continued DDoS attacks in the data center they were hosted in. No one is immune to the ‘Bad’ Internet, and script idiots out there….”

Thanks…Too much work to go somewhere else and end up in the same boat.

I guess like Chief Brody said in the movie Jaws:

“We’re gonna need a bigger boat!!!”

TO CEO of DH

website is real business and especially when there are ecommerce sites running and orders taking place of $ thousands of dollars per day least on my all websites. so its shameful and pathetic for you that you cant even able to give any quality services.

MAN its BUISNESS !!!

NOT A JOKE !!!

SO EITHER FIX THIS DAMN THING OR LEAVE FROM WHERE YOU COME.

ALL IS HAPPENING SINCE MANAGEMENT WAS CHANGED. BEFORE GUYS WERE TOO GOOD AND VERY RARE LIKE FEW TIMES IN YEAR THERE WAS OUTAGES BUT NOW OUTAGES ARE LIKE ON WEEKLY BASICS.

SO PLEASE REPLY TO THIS POST. AND TELL TO ALL DH USERS WHY YOU CANT ABLE TO MANAGE SIMPLE HOSTING COMPANY???

TODAY YOU GUYS EVEN CLOSE LIVE CHAT. YOU ARE THE FIRST HOSTING COMPANY I KNOW THAT HAS NO PHONE NUMBER TO CALL IN …. IS THIS A JOKE????

I AM WITH YOU FOR OVER 5 YEARS AND BRING HUNDREDS OF CLIENTS AND RECOMMENED THIS DH TO HUNDREDS.

IF DH GUYS HAVE SOME BIT OF HUMANITY YOU WILL EITHER CLOSE IT DOWN OR RUN IT LIKE GODDAMN REGULAR HOSTING COMPANY

REGARDS
NEO

I’ve been with DH since 1999 (14 years!). + lots of experience of other hosts DH have always been best of the shared hosters (not perfect but you should hear about some of the others). But – this last year has been the worst of the 14 without a doubt. Hope they sort it soon.

Didn’t realize these comments are on West Coast time. It’s 5:15pm here in the east, I thought the problem had been lasting for 5+ hours. My mistake (even if it’s taking longer than usual).

Guys, it’s been weeks that some of us have been reporting unexplained occurrences of our sites either being down or beyond slow. What was DH’s response, it’s all are fault. Now, we see everything crash AGAIN. How about paying attention to the signs? What about refunds? What about accountability?

We have always been big fans of you guys but when dh status is my top hit on recently viewed, something isnt right.

I wish Dreamhost could host my sites where they host dreamhoststatus.com.

@MARK

DO YOU KNOW ANY OTHER SUITABLE HOSTING COMAPNIES THAT HAS CUSTOMIZED PANEL. I AM ONLY STICKING WITH DH DUE TO THIER CUSTOMIZED PANEL. IF YOU OR ANY ONE KNOWS ANY OTHER HOSTING COMPANY THAT PROVIDES FEATURES LIKE DH PLEASE LET ME KNOW SO I CAN LEAVE THEM FOR GOOD.

Faster please, I have one of my main sites that is a url redirect only and that’s down….

@neo

As I said earlier, if your sites are truly taking $000s per day, then you should have redundant hosting with multiple providers

@neo, what do you mean by ‘customized’

I really dont want to leave DH, for the past 4 years worked awesome. Now, it seems to be a different host, dont know why. Give us back the trust. Really, I want to stay, give us back the trust we had.

Several posts above (including a previous post from me) have requested recommendations for other web hosts, but I can’t seem to see any replies offering those recommendations. Is DreamHost taking down any responses that point people toward other hosts?

Error: Database connection failed.

It is possible that the database is overloaded or otherwise not running properly.

The site administrator should also check that the database details have been correctly specified in config.php

who has been receiving the above message for weeks?

there is always go daddy… but there lies the dark side

@Rick

Posting a list of alternative provides is counterproductive, I doubt they are being taken down

You need to think what features you need, and how much you want to pay

I have seen recommendations for Hostgator, Bluehost, and others, but Im not convinced.

I looked at Hostgator and decided it was not for me

You’re KILLING ME guys! What’s the hold up?????

hahahah i have been reading comments and wanted to click on usernames to view websites…

but right… they are all down. funny. but not funny.

I am so upset right now. I guess I keep posting because I have to vent. This is the WORST!!! DH the worst think you can do is leave your customers feeling powerless!!!!

May 13th, 2013 at 2:28 pmMonkey Phalice! Says:

opened a support ticket, only to be told my site is up…..because they checked in the 2min window it was actually up… try again now please DH INTO THE BREACH!

@vasylko

It may only be dynamic sites that are down

Down !! Down !! Down !!

My business is most important… obviously! But, if I wanted to move to a different company (server), how can I get my Domain Transfered… if I can’t sign on the DH? Hmmm!

Please update expected fix time ASAP.

@DivTec

If your domain is registered through DH, and is hosted by the DH name servers, then admittedly you may have an issue right now

If your domain registrar is not DH, then it is an easy matter to change name-servers, and host somewhere else

Have been with Dreamhost for longer than most of you. And they’ve good overall. Be patient…this will pass soon!

since I cant get ANY work done…. and being a long term customer (2007) having experienced this getting worse by the week…

I have been using my afternoon to shop for new hosting….. one thing I have founf…and its pretty annoying… is since dh is using a custom interface, you can not export your hosting account like you could if it were on cpanel….

if you have the 50+ sites I have…. that turns moving to a new host a nightmare of a time suck…. but at this point one that is worth doing…. this is getting worse than ever before…. and the tech folks just flat out lie & try to convince you … its something you did, rather than admit that dh is falling apart… (not their fault)

I design & host sites here, and Im sick of my clients complaining about the downtime & Im tired of making excuses

So…for those of you like me that are just done… here are a few of the new hosts I have been looking at today….
http://www.webhostingbuzz.com/
http://asmallorange.com/

These 2 seem to offer a variety of account types… reseller vps… shared… at fair pricing
from what I can see the feedback on their forums if more good than bad

there is always some bad… the trick is getting the big picture from it… I have the big picture on dh… had it for a while… just wasn’t willing to lose all the time its going to take to make the move

btw… I have at least FIFTY …..site is up…site is down.. site is up…site is down emails in my inbox so
far today

time to simply vote with my dollars… and bail

apparently my last comment is awaiting “moderation” because I mentioned the 2 alternative companies I found while shopping for a host…since I cant work

There was a major outage last year around the same time which lasted over 3 days (72 hours)!! Let’s hope it’s not going to be as long this time.

Of course, this failure is annoying, very annoying! However, servers are machines and other devices such as cars also have the annoying tendency of sometimes breaking down.

Otherwise, Dreamhost provides a really great service and support.

I wish the technical team good luck, and hope they will sort the problem out soon.

C’mon guys! You used to be the best! 2013 is sucking! whats going on?

those would be asmallorange_dot_com
and webhostingbuzz_dot_com

I am confused about the incongruous nature of the “100% Uptime Guarantee.” You say that you “begin assessing credit from the time you open a support ticket to report the problem.” And yet, when you go to open a support ticket one of the ‘tips’ to try before submitting the ticket is “Please check dreamhoststatus.com to see if there are currently any unresolved outages.” As if you are discouraging customers from submitting tickets for downtime already displayed on dreamhoststatus.com.

I think that Dreamhost should experience a financial penalty for the gratuitous quantity of recent downtime. On the other hand I also just want my websites back up. I don’t want the Support Staff to waste time reading support tickets for outages they are already aware of. Just go fix the problem!

It seems to me that if the outage we are experiencing is posted in dreamhoststatus.com then the uptime guarantee should be handled without requiring a support ticket once the issue is resolved. Otherwise Dreamhost support needs to pull double duty fixing the issue and sledging through a host of support tickets. Not to mention the fact that calling something a “100% Uptime Guarantee” is somewhat dishonest if the guarantee is void when the ticket system is inaccessible…

I’m from Brasil and I’ve got a few websites hosted in DH.. they’re all down too.

I’m definitely transferring my site. This is unacceptable. I’m going to switch to bluehost.com

They actually have people that you can call and speak to about any technical issues.

I’ve been with dreamhost for 5 years now, but lately the outages are unacceptable. No more. Especially frustrating that you can’t pick up the phone and talk to someone.

@fed up

Interesting

I will say this has led me to do some research on better alternatives to Dreamhost, and in my evaluation DH is almost always at the top of the charts for best hosting companies. I would have agreed a few years ago, but now it just causes me not to trust the others that are in the top of the charts. We are not making thousands of dollars a day on our website. Not many people really are … but it certainly does indirectly affect our business when our clients have taken our advice to host their sites with you and I’m not receiving dozens of emails about the outages. Every time this happens, we have to send out hundreds of emails to our clients letting them know we are monitoring the progress of the outage, but our hands are tied… we can’t really help them, except to manage expectations. The more this happens, the less trust our customers have in us.

I can appreciate that Dreamhost is growing and with growth comes the inability to keep up with client demands. I’d just strongly advice a systems audit and a better plan of action for future outages. It makes me sad when I see a good thing fall apart. Dreamhost was a good thing at one point in time. I hope they will find that place again. Until then, I’m continuing my search for a new hosting company.

I too am down on all of my sites, potentially losing real business. I am getting so tired of these constant outages. I’d like a one month refund on my hosting fees. 100% uptime guarantee is BS!

@Spencer

If what you say is true, then I will wait with eager anticipation for my $1 credit

It’s probably a big, cathartic purge of an old architecture, and the temp servers failed during the move, so…

I’ve always been a huge fan of Dreamhost, but I’ve had to make excuses to my clients way too many times now. They are demanding I switch hosting companies.

^ and yes. It’s the end of the day for me and I apparently can’t spell anymore. Hopefully your engineers are more alert than me. :-)

Andrew,

No one can really offer a 100% uptime guarantee (but it is possible) with cloud infrastructures, and redundant data centers. The definition of the 100% is important though

Is it 100% server availability? , network ?… something else ?

The two DH sites of mine that I knew were down have just returned at 4:40pm CST.

Total outage was 2 hrs 20 min. Not happy about that but it’s good to have them back online.

Remember that “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

@fed_up … asmallorange_dot_com is also a good hosting company, according to the reviews … but what does that really mean when DH is right up there with them. Usually even ranking higher. You don’t even see Rackspace, etc… so it makes me think the reviews are paid. The links back to DH and the other companies are probably affiliate links. So much for that. I’ll just have to start asking around, maybe even finding someone local.

@Brady Mills

RS are not really in that market space

——– Original Message ——–
Subject: Re: [xxxxxxx nnnnnnnn] New WordPress Migration is Really Slow
From: DreamHost Customer Support Team
Date: Mon, May 13, 2013 2:26 pm
To: [me]

————————————————————————
– After reading this response, please consider visiting
– the survey below to comment on its quality. Thanks!
– [URL removed]

– If the service you received from us was exceptional, please consider
– tweeting your love for @dreamhost. It’ll warm our hearts, soothe
– our souls, and get you good karma at some point down the road.
————————————————————————

Hello DreamHosters,

Please accept our apologies for this impersonal reply to your support tickets. We’re currently experiencing networking issues across all of our data center locations. This is affecting both web servers & MySQL servers, but does not affect mail servers or delivery.

You may be seeing effects of this issue as slowness or unavailability of your hosted web sites and/or MySQL databases.

Our system engineers are hard at work on a resolution, and are continuing to apply a fix for these issues across all affected machines. The most recent reports from our system engineering team indicate that this is working but we’re going to confirm and the apply the fix across the board.

We will update you via this e-mail address when new information becomes available. You may also wish to stay up-to-date by browsing to our status post on the issue, which you can find here:

[URL to this page removed]

Thank you for your patience in this matter,

DreamHost

My site has been down for exactly 10hrs, I am very disappointed that the status updates are not giving any sort of ETA on when they expect the issue to be resolved. The service provided by Dream Host in the last couple of days is not what I expected…

All – this is really frustrating – down a whole day?

@Brady Mills

you just have to spend some time on the forums on the hosts sites to figure it out, rack space and media temple blow all the hosts like dh godaddy and hostgator out of the water… totally different class of host

all you have to do is look at a link to see if its an affiliate link…there are lots of bs “top ten” lists and review sites
you just have to use a little “read between the lines” super power… ie common sence

thats how I started using dh 8 years ago….. and then it was pretty great… it is SO NOT THAT NOW…. there is ZERO comparison the the service they used to have

Thanks for your work, I highly appreciate that you are on the issue and are trying to fix it. My sites seem to be back up online now, and I hope it’ll last! :)

My site is still down :( Hope it’s back up soon :)

my websites are down again and no real way to reach you.. haven’t gotten proper updates and don’t check all the time – but makes me nervous this happens alot more than i know. A client told me, he couldn’t get on my websites. this is bad! and no notice from you until i wrote you.

@brady Mills

Your site looks up

@Lisa

You should use a monitoring service

Our site is back online…

I would be more accepting of challenges, if the emails weren’t complete canned responses and there was some REAL communication going on.

We apologize for the service interruption still being experienced. We do understand this is highly frustrating, and are hard at work to implement our individual fixes to all affected servers and to determine the root cause of this situation. A more detailed explanation and permanent solutions are on the way as soon as possible.

@Sean H

Thanks

thanks Sean. Please keep us informed.

site is back up. but keeps on hanging up when logging into wordpress .. .

@Andrew and others: Regarding our 100% uptime guarantee, details on how that works can be found at http://dreamhost.com/web-hosting/ under the “Satisfaction Guaranteed” tab at the bottom. Please request a credit via the Contact Support page in your web panel to receive compensation and emailed updates, as they are handled individually. Thanks for your understanding!

I just have a CloudFlare message saying my site is offline and not cached. Not sure why it’s not cached as it’s a static home page?! I thought this kind of redundancy was supposed to be one of the benefits of turning CloudFlare cache on?! Asides from speedup of course…
The timing of this downtime could hardly be any worse for me :-(

My sites on isle-of-wight are still barely reachable. More communication from DH would be great.

How would a credit for for those of us who bought 10 year plans (mines not up until 2018)?

Today was the LAST TIME, i have to deal with these stupid outages. Dreamhost i am gone once and for all. Taking my 100 domains and moving them as we speak. While your have had your finger up your a$$ all day some people need sites to make a living.

Sorry but these outages are happening more and more with the usual lame SORRY.

Adios you officially suck!

So, It looks like another DDoS against WordPress

All my sites are down again. I just submitted my ticket as per your “Satisfaction Guaranteed” policy.

Another day, Another botnet

@Monkey Phalice: Our apologies. We have your open tickets and will be responding to them accordingly. I appreciate your patience.

update, my sites’ http is up, ftp down

It is getting worst and worst everyday
how long will these problems continue?
my sites are unreachable whenever i try to display :(

@berkcelebi

Its the Botnets driven by unpatched, and hacked Windoze machine that cause problems like this. If it is an external attack, it is difficult to stop without a lot of $$$

6:35 and all my sites are still down

@Irrelevant88: Every hour of downtime equals a day of hosting credit, so that’s how it will be calculated, keeping in mind the particular plan you chose.

Web and Cloud Hosting Provider DreamHost Selects Arbor Networks’ Peakflow(R) SP Platform for Infrastructure Visibility and Security
World’s Most Deployed Solution for DDoS Attack Mitigation

CHELMSFORD, MA–(Marketwire – Oct 23, 2012) – Arbor Networks, Inc., a leading provider of network security and management solutions for enterprise and service provider networks, today announced that DreamHost, a global web and cloud hosting company, has selected Arbor’s Peakflow SP platform, the industry’s leading network-wide infrastructure security and traffic-monitoring platform.

Source: http://dreamhost.com/press-releases/web-and-cloud-hosting-provider-dreamhost-selects-arbor-networks-peakflowr-sp-platform-for-infrastructure-visibility-and-security/

When Godaddy went completely down late in 2012 http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/ I was sure something like that would never happen to DH. I was not only wrong, but my confidence in DH is completely gone, I gave it a number of times…

don’t be lame people!
dreamhost is a pretty good hosting service.
It’s generally up all the time.

We live in a “brave new world” where we have to deal with malicious (not political) hackers and sometimes they ruin it for everyone.

deal with it!

“What is VPS?

DreamHost partitions a physical server into multiple virtual servers that each have a protected and reserved amount of CPU and RAM resources. Unlike Shared hosting, VPS instances do not compete against each other for these resources. Therefore, a spike in resource demand among other VPS instances, will not lead to a performance decline in your website or database.”

Really? Funny as my VPS has been unresponsive off and on all day long, and I can guarantee that it’s nothing that’s running on my VPS that’s causing it.

@Cthulhu

Quite possibly, however, if there is a DDoS Attack going on against DHs network, that COULD affect any server

eek – it’s getting embarrassing for my clients and followers. Really hope this is resolved soon. :)

Avengers is still completely unresponsive. No pings, SSH connections stuck in SYN_SENT. Looks like no packets are getting in or out. I’m wondering if the problem is related to this:

http://www.virusbtn.com/news/2013/05_01.xml
http://www.incapsula.com/the-incapsula-blog/item/715-wordpress-security-alert-pingback-ddos

ahh so it is related to all the chinese IPs I saw hitting login pages on my forum. I tried to ban the ip blocks they were coming in on from making more registration attempts but at 50 an hour they must be putting a hell of a load on the dreamhost server. I’d just black hole the whole country but I imagine DH can’t do that. Though depending on the rate of incoming attempts it might not matter with hardware at the fringe :p

@kreen

I once had a customer ask for anything non-US to be blocked

@BOFH that’s probably going a little far, heh. I’ve just had nothing but problems and spam attempts from there and it’s gotten tiresome.

May 13th, 2013 at 4:14 pmDoppelganger Says:

Has anyone ever received any of the promised credit from the 100% uptime guarantee? I certainly haven’t.

Here it is directly from dreamhost.com:

“If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that!”

@kreen

Im not denying that, but it was just to move it from his cheap / crap equipment, onto my (considerably more expensive) gear.

But as you say, n a shared host its unlikely DH will do that, however, a VPS is a different matter :)

@doppelganger well to be fair it’s arguable that this seems to be a failure due to hostile behavior not a failure of their systems persay. A DDOS or other scripted malicious behavior of sufficient scale can bring many networks to their knees. especially if this thing is botting database heavy login pages. I need to remember to change all the passwords once I can login again.

@Doppleganger: If you submit a ticket requesting credit for the downtime, please know we will provide you with the credit owed.

In the future, it would be very useful if WordPress instances could be hosted on separate boxes. That way, when (not if) somebody finds yet another way to make WP burn tons of processor power, they won’t bring everybody else’s sites down with them.

It was up….now it’s down again :(

WordPress wp-login.php brute force attack
Written by Jacob Nicholson
There has been a recent large scale WordPress wp-login.php brute force attack coming from a large amount of IP addresses spread across the world.

A large botnet of around 90,000 compromised servers has been attempting to break into WordPress websites by continually trying to guess the username and password to get into the WordPress admin dashboard.

InMotion Hosting’s defense plan
Our senior system administration team has rolled out a fleet wide security policy to help contain these attacks. For the time being we’re removing any public information detailing the new way we’re blocking these attacks, as the attackers seem to be actively monitoring for changes, and altering their tactics.

If your site has been targeted in this attack, these security precautions we’ve implemented may limit your own access to your WordPress admin dashboard. We have chosen to proactively protect our customers from this attack to avoid a potentially larger security problem on your account.

This issue is not isolated to InMotion Hosting, and is currently affecting multiple providers.

We HIGHLY recommend taking the following actions to secure your WordPress website:

10 recommended steps to lock down and secure WordPress
1. Use a strong password
Minimum password recommendations:- At least 8 characters total
– Mixture of upper and lower-case letters
– Numbers, punctuation or other non-alphanumeric characters
Example weak password: secret1

Improved strong password: Z#hupsZ2M4!Z

Take a look at how to create a secure WordPress admin password for easy steps.

2. Change default WordPress admin username
When installing WordPress by default the administrator user has the username of admin.

The botnet attack is currently only targeting this default username, so even having an administrator username of admin123 could signifiantly reduce the likilhood of your site being succesfully logged into by a malicious user.

Check out how to change the WordPress default admin username for security.

3. Lock down WordPress admin access with .htaccess
Utilizing a WordPress brute force plugin for this type of attack is not very efficient, and in some cases can actually lead to your site becoming unavailable due to the large amount of processing power used to attempt to challenge each and every malicious login attempt.

Instead you should rely on the information we have on how to lock down WordPress admin login with .htaccess.

Setup a secondary level password to prevent unauthorized WordPress wp-admin and wp-login.php attempts.

4. Temporaily disable CPU intensive login limit plugins
Blocking this attack with .htaccess rules is the preferred method, as login limiting plugins can not only lead to issue with triggering our own internal security rules, but they also will not be effective in this type of large scale attack.

5. Scan website for hacks, check Google Safe Browsing
If your WordPress site had been successfully compromised, a clear indication will usually be found either by a surface security scan of the website, or it will also get reported to Google’s Safe Browsing.

Scan your website with an online malware scanner like sitecheck.sucuri.net/scanner

Check Google’s safe browsing for your domain, at google.com/safebrowsing/diagnostic?site=example.com

6. Setup CloudFlare DNS level protection
Due to the large scale of this botnet attack, CloudFlare has offered DNS level filtering for this attack on all of their free accounts.

While probably not an ideal solution if you have many WordPress sites due to having to update the name servers for each domain, and then waiting typically 24-36 hours for DNS propagation. Single site owners might benefit greatly from this type of protection which should block the botnet requests from even making it to the server in the first place.

7. Backup WordPress
At this point it’s probably a good idea to backup WordPress just in case. That way, as the attacks continue, you’re ensured that you always have a good point to restore back to in the event something goes bad.

Backing up your data

Backup your website files in cPanel
Backup your database in cPanel
Restoring your data

Restore your website files in cPanel
Restore your database in cPanel
8. Update everything WordPress
To protect yourself from any known exploits to WordPress you should update everything related to WordPress:

Necessary updates to make:

Update WordPress from admin dashboard
Update WordPress theme
Update WordPress plugin
9. Clean up hacks
If your website has been the victim of a hack, you can follow my guide on how to reinstall WordPress after a hack for steps on cleaning it up and getting back in business.

10. Other general WordPress recommendations
Optimizing WordPress with W3 Total Cache plugin
Log out of WordPress admin dashboard when not in use
Limit or disable WordPress revisions
Disable WordPress autosave
Install and use Better Delete Revision WordPress plugin
Hopefully your WordPress website should be locked down and secure now, which should help prevent our own internal security rules from blocking your own access to your WordPress admin.

If you’re blocked out of your own account and were directed to this article via a warning message be sure that you followed step #3 Lock down WordPress admin access with .htaccess in order to regain access.

@dgatwood I imagine this is the same problem that my PHPBB forum was having. guess they’re not very picky?

Well, I feel better knowing that the cause is related to outside attacks.

Sorry that was another hosting companies advice.

Dreamboat isn’t the only host that has had to deal with this attack.

I am a DreamHost customer for over 5 years but it is time to go, I always liked Dreamhost but I want a company to help me grow and not to drag me down. Today I spent almost the whole day apologizing to clients and it is getting worse every week.

should I be worried about my wordpress site being hacked?

@jon heh! I imagine the bots are causing too much load to the target to brute force effectively now. It’s just good practice to change them after something like this.

Hey – we’ve been down all day! I’ve been apologizing and asking people to “wait a few hours” for you since this morning, do you have any idea what’s going on? Are you going to fix it sometime soon?

Jeeeeez…

@Carrie

or just remove the wp-login.php page

My site is still down. I’m worrying about issues when everything is fixed and I need to log into my site. What is the deal? @jon, when you mention “it’s good practice to change them after something like this” are you referencing our username and password? If there are other steps we need to take to protect ourselves, I hope DH tells us.

I just got the ‘500 Internal Server Error’ and error 404. I don’t know what to do. My site has been down all day :(

Did anything leak?????????????????????????

The fact that they have stated this is a DDoS Attack to me is different than their hardware. I wish they would have said something sooner than just waiting so long for this statement would have helped lots of people with their frustrations at DH. My sites are currently up and down all day so I will assume this is going to take some time to get resolved.

at least they are keeping us posted. no point in bitching i am sure they are doing their best. gambatte

@ashley this is a horde of remotely controlled computers trying to brute force logins to every server on the block from all corners of the internet. They’ve applied so much pressure the server buckles. Suppose 90,000 people lined up in front of every door in your neighborhood to each try one key of all possible keys then move to another line if it failed. Not only would you probably be at the back of the line for an hour but the line itself will inhibit the proper flow of traffic around it when you get to that kind of scale. After the attack is done just change admin passwords like the advice in that infnode post above suggests.

I always talk about the excellent services from you guys….. 30 clients screaming about the services. It’s been a long time. At least post every hour to keep us calm.

@ Doug but how would I login to my wp site without the login page?

@Jonathan Burgos: Our apologies again. We’re making sure to provide you with updates on our status post every 30 minutes.

@Kirill this isn’t that kind of attack. I’d say it’s unlikely they got to try many logins before they brought the servers down from being too aggressive but only DH would know how much time they’ve been poking about. The worst I think that’s likely to happen is DH has to reset a password from something that tried to capture access. Once the hair-on-fire server admins stop the zombies from gnawing on walls I suppose they’ll have time to figure out damage.

Well, at least we know what the issue is finally. It’s odd though, I have 3 other WordPress sites on different hosts and NONE of them are showing any difficulty, in fact those servers are moving as fast as I can remember. I implemented the same exact security measures on them that I was instructed to on Dreamhost, and they’re doing absolutely fine. Meanwhile, it’s been at least 6 hours that my site’s been down on Dreamhost – maybe the issue isn’t a targeted attack on WordPress so much as a targeted attack on Dreamhost, and the techs should really be handling it better than this.

Is there any update. This is a bit rediculous. for a group that JUST launched their website and sent out a communication to visit the website and to have the site down for as long as it has been is horrific. PLEASE put more people on the problem. if i were seeing issues with other sites across the board, i would feel better, but that is not the case, others seem to be operating okay. in the last few months this is one time amongst many that the site has been down- i am beginning to think that we need to look at another hosting entity. i have heard that Dreamhost has been good, but seems to be going down over the last year or so- is this others experience?

@suz I personally have never seen my site down for so very long before, it’s otherwise just been a hardware failure every now and again for a an hour at worst. It seems someone sicced a botnet at a few places in particular but aren’t smart enough (thankfully for us) to run it properly without breaking the thing they’re trying to log into. nowadays it’s not really a question of if but when for a host to be hit. This attack may pop up for pretty much any host you go yo unless you pay a lot of money for them to be throwing stupid amounts of hardware at the problem. then it’s hardware that is likely to never be used to capacity for months at a time. TL;DR Where ‘zactly do you think you can go for cheap and avoid botnets?

Some of these comments are so bilious. I have been a DreamHost customer for 5 years, lots of my sites are down, dh should really handle this better, clients are going mad. I live in a shoe. This is the worst thing that has ever happened to me. Seriously my life is that cretinous. Meh… Get over it for flucks sake

Timmy, your comment made me think of the album Bilious Paths by μ-Ziq. Good stuff.

74 queries in 59 seconds
what the hell

And why phpmyadmin of dh don’t update to the newest version?

So, I have been with Dreamhost for over 10 years, and after all the problems we decided to finally leave. We completed the move this Friday to FireHost (amazing company).

Seems like we got out just in time!

Ion

Kreen- thanks for the response. i am sure you all want the problem fixed as much as anyone and i appreciate your quick follow-up to my email. i have checked on multiple sites (randomly) and truly haven’t seen the issue anywhere else. i will continue to check for updates and again appreciate the efforts.

Timmy- your responses are just… well, i won’t continue. some things are better left unsaid.

My website is up, but the EXEC function on php is not working… IDK what to do… That function makes the e.commerce process to work.
Does anyone have a solution? thanks….

I have one really, REALLY important website of game sales.

There are a lot of sales right now and none of my clients are able to reach the page, which is the only way to make payments, verify them, retrieve games, etc.

I’ve been waiting all day and all i see is stupid responses, when will you be responsible on the service you provide? This problem is from 1 week ago and you still do anything.. shame on you, luckily i’m going to pay another host so luckily i’ll get refunded from this service.

I have been with DreamHost since 1998. They will fix this. What surprises me is the attitude that a DDoS is something that can be fixed in a snap of the fingers. I know many of you are not saying that, but some of you are, and those of you who are, need to know that this is a serious event. If we have an X-Wave Solar event in the next two days, it could compound the problem, by limiting everybody’s satellite options (if you have those server options). So be cool and let these pros do their job. Besides, if they can’t, frankly, I’m not sure anybody can.

All you people complaining should realize you are paying for a shared server hosting plan. if you want to cry ass about your site going down, pay $150+ a month for fault tolerant hosting; they call it 99% uptime for a reason.

I want my three sites where dreamhoststatus.com is located!!!!

@Carrie. The question is how would a brute force attack to that page login? Might solve the problem with this kind of attack. Just copy the page to your local pc, then remove it from the server. When you need to login just upload.

9 hours and still down. Can someone recommend a good alternative to dreamhost? They’ve screwed us beyond any reconciliation

@ralphTT like dick said, once a botnet starts it’s not a small thing to get it under control assuming it’s attacking in a pattern where you can do that easily. You can go somewhere else but if this kind of botnet attacks a shared host you’re going down. If you want to have a chance to survive a botnet attack of any real scale you have to pony up for that 150$ fault tolerant hosting that’s got the extra hardware on hand. :) If this was actually a problem with hardware issues I’d have more problem with it. I don’t recall any internal problems that ever took anywhere near this long. The main reason this is taking so long as the majority of the problem isn’t internal and so not under their control. I’m sure they have more than qualified people that are working to mitigate the attack.

@Suz — i’ve noticed more downtime periods over the last few months, but this is definitely one of the longer ones

@weed come to think of it I’ve noticed the spam attempts and spurious registrations picking up in the last few months as well. I imagine these two things may be related?

I recommend HOSTGATOR i been working for 2 months since every week it’s the same. I been working so fine and i recomend a lot.

Any updates?
I was getting the identical emails from support over and over today. My site was down on the 11th and I thought the issue was resolved. Now there are DOS attacks. This is frustrating.

It’s a cyber battle, folks!
(My website and my blogs are down as well…)

Updating the problem, my MySQL server is Down.

http://www.huilbook.com/dh_phpmyadmin/datahuilbook.huilbook.com/

what happen? ….

Don’t you afraid your status blog got DDOSed?

As Carrie suggests, use htaccess to only allow access to wp-login from your IP address. Also use a plugin like WordFence to stop attackers. You can also add a .htaccess to your wp-admin folder limiting access your unique IP address. Of course update for a non “Admin” username, a strong password, WordPress itself and all plugins and themes. Cloudflare is showing a big jump in attacks yesterday and today. I do have other wises on MediaTemple and BlueHost and they are running fine. BBS reports on HostGator’s solutions here: http://www.bootstrapbusinesssolutions.com/blog/wordpress-websites-everywhere-are-under-attack/#.UZGPOJWDpZ0

There is one other thing we need to consider. News articles on this kind of thing stress that there is no zero point – that is, they have no single origination place. In cases this complicated and this broad, there are usually at least ten people, and as many as a hundred, who come together, almost like a flash-mob, to originate a DDoS. What makes it worse is that even after the first strike, a residual force continues to make trouble through secondary false ips. What I’m getting at is that, depending on WHO’S doing the cracking, this could take quite a while and have international repercussions, assuming the Justice Dept. gives a damn about us little people, or the Feds let the CIA do their job.

My sites aren’t down but I can’t log in to wp:

“Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.”

What is this “attack” that DreamHost says they are having?

What can we do to help?

Randy.

I have been with Dreamhost for more than 8 years. I have literally lost websites due to hacks and attacks and this is just about the final straw. I thin it is time to look for a more secure service.

My sites are back up but as someone else said, I cannot log into WP. When attempted, I get error stating, “he server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.”

OMFG! THE PHONE IS RINGING OFF THE HOOK WITH ANGRY CLIENTS. I AM LOOSING ONE HUNDRED THOUSAND DOLLARS A MINUTE! HOW DARE THIS HAPPEN ON MY SHARED HOSTING PLAN THAT I RARELY EVER PAY FOR WITH ALL THE CLIENT REFERRAL MONEY COMING IN!

I DEMAND THIS BE FIXED whenever you can because i know you are working very hard to fix things! :)

– DH Cust since 2002

OUR WEBSITE is down and we not able to get it at all…its a business and we need it ASAP back!!

LOL @Mike Hunt

I really don’t get the attitude of some folks that because it is an inexpensive hosting plan, no one has the right to expect not to have considerable down time. Mike, idk if your comment was directed at me since I had just posted but if so, I would point out that I stated this is not the first issue I have had. I have put huge amounts of time into websites that were hacked or otherwise damaged in addition to down time. And while you might make tongue-in-cheek comments about people losing money, to some folks, they need the revenue their web sites generate. If DH wanted more money they would charge it I suppose and as long as I am paying the asking price for a service, I expect it to work reasonably well.

Any updates please? 10 hours down :(

These comments are depressing

@jenny If all downtime was equal, certainly. This kind of downtime due to hostile activities you have to pay for to control ahead of time because it’s decidedly non-trivial. Accounting for this kind of sudden load would involve a lot of extra hardware that implied access to won’t come cheap. I’m starting to get site access again so it looks like they’ve got it under control for the large part. this kind of thing is likely going to have them running defense for a while yet though.

I started with Dreamhost in 2005 – not as long as many people here. Still, it’s a long time to be a customer. I took one of the suggestions above and opened a hosting account at Media Temple. At $20 for the standard hosting, it’s more than 2x the Dreamhost cost. Even though I’m only hosting my personal blog, I would rather it be up and running. Over the past 30 days, pingdom reports that my site went down 24 times for a total of 19h 30m. That a 97.3% uptime. Also, over the past year I’ve seen my average ping go from 1,038 ms to 6,800 ms – which bothers me more than the uptime.

All in all, Dreamhost is a good host for the price… just not great.

@kreen your point is well taken. Just frustrated as it is a build up of issues for me.

Check out my blog about hosting http://www.acmedata.in and yes it is up. Because it is not on Dreamhost lulz

This is a brute-force attack on wordpress login pages, so why do you keep complaining while DH buddies trying to fix somethings ? No one enjoying downtime but you all should enjoy not being defaced huh ? Some ridiculous comments about business, service and losing money there’s a concept named karma out there hope you give your clients better than this. P.S: I am neither endorsed nor paid by dh maybe you guys are from other hosting companies leaving crap comments LOL

My sites are back :)

@acme Looks down to me matey.

Mine are up, but WP logins are still answered with a 503.

WP logins are still answered with a 503.

@jordan. yeah karma. Spoke too soon.

ah its up. Its being hit with a lot more traffic suddenly. which is weird. I still use a micro instance though.

well well. my webmaster support forum is back up http://forum.bingesite.com/ its still on DH. If I had members I would move to linode.

@hasan From what’s been said, this could have all been prevented with a simple ratelimit implementation. Maybe I’m wrong, but I know for a fact Hostgator, and other big webhosting companies already have this filter in place.

I used to love dreamhost and recommend them to everyone, never again. so long dreamhost.

i used to be a dreamhoster like you, then I took an arrow in the knee

-sorry i just could not help myself

Man, this stuff is making me super paranoid. I already have WP Better Security, would you guys recommend adding Wordfence or Bulletproof security plugins as well?

My sites work ok now… But the normal perfonance is gone. I have 20 to 50% more of lag. A page with 2.2 seconds of charge, now have 5… I don’t care if happen over some hours, but i bet that is permanent.

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

May 13th, 2013 at 8:57 pmMelanie Reed Says:

Dreamhost, Sorry no deal. This type of WordPress DDoS wp-login attack was reported almost a month ago (April 15) by CERT, so you knew about it – or should have known and put up a defensive layer.

National Cyber Awareness System
WordPress Sites Targeted by Mass Brute-force Botnet Attack

Original release date: April 15, 2013

US-CERT is aware of an ongoing campaign targeting the content management
software WordPress, a free and open source blogging tool and web
publishing platform based on PHP and MySQL. All hosting providers
offering WordPress for web content management are potentially targets.
Hackers reportedly are utilizing over 90,000 servers to compromise
websites administrator panels by exploiting hosts with admin as account
name, and weak passwords which are being resolved through brute force
attack methods.

CloudFlare, a web performance and security startup, has to block 60
million requests against its WordPress customers within one hour elapse
time. The online requests reprise the WordPress scenario targeting
administrative accounts from a botnet supported by more than 90,000
separate IP addresses. A CloudFlare spokesman asserted that if hackers
successfully control WordPress servers, potential damage and service
disruption could exceed common distributed denial of service (DDoS)
attack defenses. As a mitigating strategy, HostGator, a web hosting
company used for WordPress, has recommended users log into their
WordPress accounts and change them to more secure passwords.

US-CERT encourages users and administrators to ensure their installation
includes the latest software versions available. More information to
assist administrators in maintaining a secure content management system
include:
* Review the June 21, 2012, vulnerability described in CVE-2012-3791,
and follow best practices to determine if their organization is affected
and the appropriate response.
* Refer to the Technical Alert on Content Management Systems Security
and Associated Risks for more information on securing a web content
management system
* Refer to Security Tip Understanding Hidden Threats: Rootkits and
Botnets for more information on protecting a system against botnet
attacks
* Additional security practices and guidance are available in US-CERTs
Technical Information Paper TIP-12-298-01 on Website Security

Relevant URL(s):

Still getting the 503

So many people are complaining that they are losing business, but any smart business would not rely on a budget host like Dreamhost. This is what happens when you cut corners and don’t put in the big bucks to secure your cash flow and this is why I laugh at your DH hate comments :)

May 13th, 2013 at 9:10 pmLow Tolerance Says:

Peter you sound like a witty gentleman indeed (not). If you have nothing helpful to say… you know the rest.

Your low tolerance has backfired upon your own comment which does not appear to be helpful :)

I was happy with DH but past few months it has failed to deliver satisfactory service. I might think about moving elsewhere.

still cant login the wp-admin

Reading some of the comments here, I wonder if you actually saw the title of this post which included the words “External Attacks”

So how exactly is it DH’s fault that the systems are being hammered by brute-force attacks tartetting WordPress?

How about you all take a chill-pill and target your wrath and anger toward the pox-ridden miscreants who are firing off the attack rather than the peeps at DH who are working their arses off to fix it.

Just saying.

Hey guys, just adding another one to the mix. I’m unable to access the wordpress control panel for one of my sites, 11thlddems.org

@lisa The vulnerability was exposed over a month ago. Other WordPress sites I run, luckily hosted with other hosts (in Australia), are not feeling the impact of these attacks at all – most likely because their security measures are up-to-date. This is why people point the finger at DH – because they should have been better prepared for such an attack.

@Peter – all the preparation in the world won’t always stop a brute-force attack.

Imagine, this attitude is like blaming a soldier for not being able to defend against an attack from another country. Is it cathartic? Possibly. Is it fair? Not particularly. DH employees have not-only informed everyone of the issues, but are also posting updates on progress, something that’s pretty much absent from other hosting companies I’ve been with.

Let’s blame the hackers who have started the attack rather than the peeps working to solve it.

Gee Whiz! 100% Uptime Guarantee… I’m having site down issues every single week since DH moved my server to their new infrastructure. Every time I’m logging a case they are saying something like “it’s your php scripts”. Guys, without php I’m still having problems loading static html pages and images. I’m thinking about moving away from DH. Too many problems and too unreliable service.

@Peter, what hosting service in Australia are you using? Thinking about moving my service close to my home. :)

@lisa personally I believe a majority of the blame does fall on those with malicious intent, although there is a middle-ground where DH could have taken measures to be more well-equipped for such a common attack.

In the end people will kill the messenger because it’s simply human nature to do so. Some can control it, some can’t control it but it’s something that we all go through one day.

still

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

@Ilya Genkin I’m with WebCity! They are great, have been with them for about eight years now and they VERY rarely experience problems!

Yeah a couple of my pages hung up, so I checked the status page. Do you people whining even know that this is an attack on WordPress, not a Dreamhost hiccup? Everyone is affected, everywhere. This is how your host is responding. Be glad they did anything, gladder yet they told you they were doing it.

YAY! I’m back! Thanks DH…

It may be an attack on WordPress, but if DH sets itself up as the ultimate wordpress host, you expect them to be able to host it. 10 hours after it started I cannot upload to my non-Wordpress site. This is not so much blaming the soldier for being attacked as blaming the sentry for falling asleep.

@CGHill, @oks, @Jenny, @Robert, 503 errors are being worked on. Keep your eyes on this post. If you still have issues with 503 errors upon logging into WordPress *once it’s been marked as fixed*, please contact us via the Web Panel >> Contact Support page.

@Laura: The mentioned WordPress plugins will help in these types of attacks. Another recommended action is to maintain a secure password and keep WordPress updated.

@OscarCo: You may still experience some lag in site loading times (or even downtime) as server resource consumption remains high. Some hosts are still in need of our fix.

@Melanie Reed, @Peter Issa, : We have implemented fixes against these attacks in the past month or two along with collaborative efforts from other hosting companies and security firms. The attacks are highly evolving, so sometimes sudden changes in attack vectors can render previous fixes ineffective. To reiterate, measures WERE taken against the attack you’ve mentioned, but consider this attack an evolution.

Need more status on Twitter.

@DH Brian Thanks for the clarification!

More DDoS. I’d ask Arbor for a refund.

Just FYI it’s not just WordPress that’s affected. I have three domains and they are all down together (all hosted on the West Coast). Only one of them is WordPress. My sites were up again for about 45 minutes, until about 10 minutes ago when they went back down.

I’ve been trying to download a large (6.2 Gig) backup file off of my dreamhost account via sftp. I guess I picked the wrong day to do so, because it keeps timing out, getting disconnected, and not allowing me to resume the download, likely because of their network blocking changes.

I’ve been running Better WordPress Security ever since my site got hacked a couple months ago, and I’ve been seeing 20-40+ attempts to login to “admin” daily, but they dropped to zero for a couple of days there. I’ve got most all of the recommended BWPS protections set, but I can’t check anything because I’m locked out. Hopefully someone is looking at each site and seeing that some of us are actual users and not part of the bot attack.

@Fred: You are right in that all domains have the potential to be affected by rolling downtime or slowness due to the nature of this problem. The WordPress issues were resultant of a widespread fix, but those should now be fixed. HOWEVER, WordPress domains might still be slow or due to server load, though MOST hosts are now considered stable. I’ll update the post now with the latest information!

Hosting is very low connection for my websites.
:(

We have been customers on #dreamhost for over 10 years now.

The last 3 months have a hell with regards to several sites crushes on daily basis and clueless support (some of your “experts” do not even know the different between cgi perl and php – NOT joking..)

I seriously think you guys should check- whether some of your new (?) employees came from GoDaddy..

I’ve been having daily ‘downtime’ lately but today is worse it’s been down for hours. Very disappointed.

Site is down again. Can not perform updates or check orders. DH no good.

All my sites are dead, not a cordless among them, what is going on there, this is ridiculous.

All my sites are dead, not a wordpress among them, what is going on there, this is ridiculous.

@Dave Reid, please see the latest update providing context. The outage does not only affect WordPress domains.

@On Elpeleg: your support request is not related to this outage. You should now have a response to your email support request.

@Jordan Rate limiting might help, but it can be problematic for two reasons:

1. Different services have vastly different needs with vastly different workloads. A rate limit that is appropriate for a WordPress login (with high overhead per request) would completely destroy the functionality of a website that uses a RESTful API to makes lot of lightweight requests for resources in a short period of time by preventing legitimate requests. A rate limit that is appropriate for the RESTful site would allow even a relatively lame attacker to completely bury a WordPress instance.

2. If a botnet has tens to hundreds of thousands of machines in it, even with an extremely efficient web server (read “not Apache”), each machine need only make a request every few seconds to kill a server running a heavyweight service like WordPress. With a less efficient setup, a complete DDOS might require only one request per source IP every few *minutes*.

I’m not sure what a good solution would look like, but odds are, rate limiting isn’t it, unless you are seeing the same IPs hitting multiple machines at the same time and can somehow manage to correlate those requests to generate a list of problem IPs and block the initial SYN packets from all of them using ipset.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking issues affecting US-West Data Center (Irvine, CA) (RESOLVED)
9:13 am

Networking issues affecting US-West Data Center (Irvine, CA) (RESOLVED)

Posted (May 12th, 2013 at 9:13 am PST) by danwy

Some of our customer in our US-West Data Center (Irvine, CA) may be having some connectivity issues at the moment. This includes web and mysql services, but no mail services are affected. We are investigating the root cause and are working towards resolving it. We are very sorry for the inconvenience this is causing, and will post update as soon as we have more info.

To see if you may be affected, please check your data center location in the DreamHost panel.

Update May 12th 3:44pm PDT: We now consider this incident resolved; we’ve been monitoring connectivity for the past few hours and have not observed any further issues. If you’re still having problems with your web and mysql services, it’s unrelated, so please contact support with the details, so we can assist you.

Update May 12th 1:35pm PDTWe’ve been keeping an eye on things for the last hour and things have been running smoothly with no packet loss. We believe this issue is resolved at this time, but will keep this post open for a bit longer while we continue to keep an eye on things. If you continue to have issues, please contact support to get your problem looked into.

Update May 12th 12:35pm PDT: We were able to narrow down the cause of these issues to some exploited customer code. Our Security team continues to scan the affected servers to make sure that we’ve caught it all. The packet loss caused by this issue has dropped again at this point, but we’ll continue to keep a close eye on the situation. We will leave this status post open a while longer until we’re confident that we will not have any further issues.

Update May 12th 12:10pm PDT: This issue has come up again and is causing intermittent issues for a small handful of customers. Our admin team is already investigating it again and we’ll post more information when we have it.

Update May 12th 9:55am PDT: Our networking team was able to find the cause of the packet loss in the datacenter and have resolved the issue. We’ve tested the effected servers and have confirmed that there are no more signs of packet loss. If you continue to have problems with your services, please contact support for further assistance.

Update May 12th 9:40am PDT: We’re still investigating this issue, but don’t have anything new to report at the moment. As soon as we have more information we’ll update this post.

This entry was posted on Sunday, May 12th, 2013 at 9:13 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

32 Responses to “Networking issues affecting US-West Data Center (Irvine, CA) (RESOLVED)”

Guy, any update? My site is sooo slow now and sometimes doesn’t load at all… It is not acceptable.

It’s turning into a serious habit … I’m literally on the brink of leaving DreamHost.

Really guys, “may be having connectivity problems”? You have an outage, It has been going on for almost 1 3/4 hours now. How about a meaningful status update so we can make some decisions?

My mysql server has been on and off all morning.

This really is getting ridiculous. This is a weekly occurrence now.

And how come it always happens to Irvine, CA – my other websites in other DCs are usually fine and those are affected very frequently! Is it always the first one you are trying new things? May we ask to be moved to another one?

Yes, my Database and XML is 100% OFF LINE, not connectivity at all. Please fix as soon as possible. This seems to be happening every few months. I’m paying for dedicated hosting and feel like you are that dedicated. Please fix as soon as you can.

Hey guys, the problem is not resolved!

This is not resolved. ALL of my Database is DOWN. All of my customers have zero access to their galleries and files.

This is NOT fixed. Please fix immediately.

Andrew

The problem is not resolved! This is getting ridiculous. This is a weekly occurrence now.

Still down here, as well. Unacceptable that you would call it “resolved” without verifying that it actually is resolved for your customers. If it’s a case of “we’ve resolved it on our end and it will take some time for things to come back up” you absolutely MUST tell your customers that.

It is NOT resolved for me as well. It was working for 0.5 hour and is still down now!!!!

Still have issues!!!! Hello DH…. is anyone home????

Getting “Error establishing a database connection” on all wordpress-using sites

It just went down for me a few minutes ago, so I doubt it’s still resolved.

Here we go agin.

Database connection error (2): Could not connect to MySQL.

Definitely not “Resolved”.

Hi, my site appears to be down also. After being hacked recently, I’ve created a new site and it is not available again. It seems like php is not working, since the pure html files are loading fine. After experiencing awful hacking, this situation is irritating. Could you post info about what’s happening, are you working on fixing this issue?

OK, we’ve had a serious loss of data. Anyone else have data lost of their server?

It’s after 1:00 pm CDT and I can’t do anything. :-(((

this page has been taken off the DH status main page, nice move!

suggest everyone tweet @DreamHostSimon and @DreamHostCare with their thoughts in this RESOLVED outage

Two sites down with database connection errors…. Not good, I’m in the middle of a sales promotion campaign!

Still getting a 404 with my website… at 1:37 pm PST

2:10 on the west coast. We’re still getting “Error establishing a database connection.”

My site is still down with Internal Server Error. Panel shows “US – West (Los Angeles, CA)”.

My site has been down now for at least a full day! This is really getting bad. Please, let us have some updates soon…

My site was up intermittently today, and is down now. 11:14 pm edt

Was up for 15 minutes and is back down.

8:52 am on The 13th. The site just went down.

9:15 am on the 13th, the site is working

I’m a happy camper…why?…because I’m no longer hosting at Dreamhost! :D
Too many outages just like this one!
good bye

stop complaining and do something about it!
whining does not benefit anyone.

All sites still down. Was working earlier, just went down a few minutes ago.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network maintenance in US West Data Center (Los Angeles), May 14th 6pm (8hrs) (RESOLVED)
8:02 pm

Network maintenance in US West Data Center (Los Angeles), May 14th 6pm (8hrs) (RESOLVED)

Posted (May 11th, 2013 at 8:02 pm PST) by andrea

We are conducting network maintenance in our US West Data Center (Los Angeles) on Tuesday, May 14th at 6pm PDT. This maintenance involves moving networking routes to brand new equipment, while directing traffic through thoroughly pre-tested and staged backups.

The maintenance is set for an eight-hour window, and we estimate most of this to be not customer-facing.
To see if you may be affected, please check your data center location for mail, web and mysql services in your DreamHost panel.

We will post updates as the maintenance progress and will be monitoring comments. If you have any questions or concerns, please contact support with the details, so we can assist you.
We appreciate your continued patience while we improve our networking services.

Update May 15th 1:40am PDT: The IPv6 tests didn’t work as expected, so we are moving that network back to its original location. For now, the maintenance window is done. We will move IPv6 at a later date. We appreciate everyone’s patience while we performed this maintenance.

Update May 15th 1:20am PDT: We are currently testing an IPv6 network. This should cause some downtime, but _only_ on IPv6. IPv4 (normal) network traffic should not be affected.

Update May 15th 1:10am PDT: We’ve moved over the last remaining networks. These moves have all been for IPv4 functionality, we have not moved any IPv6 networks at this time. We are now assessing moving IPv6 networks. We’ll update as soon as we have a consensus.

Update May 14th 12:40pm PDT: A few more networks have been moved. We had a slight hiccup during one, but were able to resolve the problem. Now moving on to a few more. Again, only a few seconds of downtime should happen here.

Update May 14th 12:30pm PDT: We had a brief internal hiccup, however, we’re back on track now. Moving several subsets at once now, expect a few seconds of downtime in the next few minutes.

Update May 14th 11:34pm PDT: Next up, we’re doing a network that can affect shared and VPS web services. We don’t expect more than a few seconds of downtime!

Update May 14th 10:55pm PDT: That went smoothly with very little downtime. We’re regrouping, and making some adjustments now, we’ll update once we have the next network ready.

Update May 14th 10:34pm PDT: We’re ready to move our first customer facing network. We anticipate small blips of downtime lasting a few seconds. The first move will affect our sub5 mail cluster’s network, then we’ll move on to sub4 closely after.

Update May 14th 9:30pm PDT: Back on track, we’ve fixed the internal issue we encountered and are now moving on to re-routing internal networks over to the new hardware. There will be no customer impact during this change.

Update May 14th 8:09pm PDT: We’ve come across an issue during our internal staging and are pausing things for a bit, while we work on this with a vendor to resolve it. We’ll update when operations continue.

Update May 14th 7:25pm PDT: We’re about to start testing internal, non customer facing networks to move over to the new hardware. No downtime is expected.

Update May 14th 6:30pm PDT: We are now bringing one of our first links in the network chain up. This is not customer facing so there will not be any downtime.

Update May 14th 6:00pm PDT: We’re here, and ready to start! Right now we’re gathering the troops so we can begin. We’ll update when things get underway.

This entry was posted on Saturday, May 11th, 2013 at 8:02 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Network maintenance in US West Data Center (Los Angeles), May 14th 6pm (8hrs) (RESOLVED)”

A ver que nos espera.

Que salga todo de 10!

my website had some issues, but everything is fine now. good job, thank you…

Looks like my sites are hosted on the US West Data Center. Is this why my site has been receiving 503 Service Temporarily Unavailable past 6hrs?

@HD-Trailers.net, 503 errors would mean you’re not related to the outage. I have one of my techs looking into your problem. They’ll email you with more info.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues for subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)
4:26 am

Connectivity Issues for subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 11th, 2013 at 4:26 am PST) by andrea

Some of our customer in our US-West Data Center (Los Angeles, CA) may be experiencing connectivity issues. This includes web and mysql services, but no mail services are affected. Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses.

To see if you may be affected, please check your data center location in the DreamHost panel.

Update May 11th 5:10am PDT: We now consider this incident resolved; all connectivity issues have been been handled and haven’t experienced anything related during the monitoring period. If you’re still having problems with your web and mysql services, it’s unrelated, so please contact support with the details, so we can assist you.

Update May 11th 4:50am PDT: We believe we have identified and resolved the root cause for the incident, and services are coming back up now. We are continuing to investigate and monitor the situation and we’ll resolve this status post when we’re completely in the clear. We appreciate your continued patience!

This entry was posted on Saturday, May 11th, 2013 at 4:26 am and is filed under Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Connectivity Issues for subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)”

I’m also experiencing problems although servers are @ US – East (Ashburn, Virginia. Web studio-echo.com loads very slow.

I’m also hosted at the Virginia data center and have been experiencing a downtime for the past few hours.

Thanks, am glad you are looking into this. We are also in the east, Ohio.

I can tell I’m affected because my sites have been up and down all morning.

Afternoon here in Ireland, site been down for the last few hours, Hopefully ye will resolve it shortly

We can confirm that we aren’t having any issues with our Virginia data center, and the above incident was short-lived, so we’ll be looking into your individual domain status shortly and get back to you.
@Mike, please contact support through the DreamHost panel with the details if you haven’t yet, and we’ll be happy to help!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web Service Disruption Affecting Subset of Customers at US-West (Los Angeles) and US-East Datacenters (RESOLVED)
4:37 pm

Web Service Disruption Affecting Subset of Customers at US-West (Los Angeles) and US-East Datacenters (RESOLVED)

Posted (May 10th, 2013 at 4:37 pm PST) by Elizabeth

We are investigating intermittent service interruption affecting a small number of web servers at the US-West Los Angeles and US-East Virginia data centers right now, but we do not have an estimated time to solution yet.

If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

We will let you know as soon as we have more information what the problem is and what we are doing to alleviate it.

Update May 10th 6:45pm PDT: We are no longer seeing problems with memory-related outages on any of our shared hosts. We are still investigating the root cause, but at this time we are resolving this issue. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Update May 10th 6:05pm PDT: Most of the impacted servers have returned to normal load and function but we are still working hard on the remaining machines. Thank you for your continued patience!

Update May 10th 5:45pm PDT: We are still working feverishly on a resolution but server performance is improving on many of the impacted servers.

Update May 10th 5:14pm PDT: Currently a handful of servers (about 1-2% of the hosts) at two data centers are experiencing service disruption. We are still investigating but so far have seen nothing to indicate an attack or widespread networking issue (the primary symptom is that they are running out of memory). This is only impacting web services (our Panel, customer MySQL and mail services are not effected). Thank you for your patience and understanding as we get to the bottom of this.

This entry was posted on Friday, May 10th, 2013 at 4:37 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Web Service Disruption Affecting Subset of Customers at US-West (Los Angeles) and US-East Datacenters (RESOLVED)”

Indeed appeared unable to access, dreamhost is responsible earnestly. You are great

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Secure certificate changes coming for IMAP and POP on homiemail-sub4 and homiemail-sub5 email clusters on May 14th (RESOLVED)
3:20 pm

Secure certificate changes coming for IMAP and POP on homiemail-sub4 and homiemail-sub5 email clusters on May 14th (RESOLVED)

Posted (May 9th, 2013 at 3:20 pm PST) by andrea

We are happy to report that we are getting closer to changing the secure certificates for IMAP and POP on all our email clusters. As you saw, it’s already rolled out on homiemail-master and homiemail-sub3 and we’re working on the final steps to make it happen on homemail-sub4 and sub5. We’re estimating the rollout date to be next Tuesday, May 14th. We will update this post when the changes are imminent and if there is any changes in the rollout schedule.
To check your mail cluster, log into your DreamHost panel and click your Account Status on the upper right hand side.

If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, here is a quick notice for you to prepare: you will likely run into a warning when you log into your email client again (after the change!), stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that.

For now, this is only affecting IMAP and POP access. We’re still working on and testing the options for the SMTP secure certificate change, and we’ll post again when we get closer.
We will post again when we roll this change out to the other email clusters, and when SMTP is ready to go as well.

This is good news for those trying to use gmail as POP or IMAP to access their DreamHost email, as they will be able use that service without warnings and problems.

Please contact support if you have any questions.

Update May 16th 10:15am PDT: The changes have now been applied to all of our mail servers. If you have any issues connecting to your email account within your email client or sending/receiving please make sure you make an exception for our new certificate. Once you have done so, you should have no issues connecting or sending/receiving. In some cases you may need to delete our old certificate in your email client and accept our new one and you should be all set.  If you have any questions or concerns, please contact support, so we can assist you.

Update May 15th 3:49pm PDT: We are now rolling out the changes to the above secure certificates. Our support team is standing by would you have any questions or concerns.

Update May 14th 2:19pm PDT: Due to recent events, we’re rescheduling this email cluster update to May 15th. We apologize for this, however, we feel its best to delay to ensure we have the focus needed to make this change. If you have any questions, please let our support staff know!

This entry was posted on Thursday, May 9th, 2013 at 3:20 pm and is filed under Announcement, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues with US-West Data Center (Los Angeles, CA) (RESOLVED)
8:09 am

Connectivity Issues with US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 9th, 2013 at 8:09 am PST) by Oscar

We are currently experiencing issues with some servers in our Los Angeles data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this DDoS attack. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update May 9, 9:00am PST:  Our network engineers and system administration team have blocked the DDoS attacks we were seeing to our Los Angeles data center. We will continue to investigate the cause of the attacks so as to prevent similar attacks in the future. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Update May 9, 8:50am PST:  Our network engineers and system administration team are still working on blocking the DDoS attacks. We are making progress and narrowing down which machines are specifically being targeted. We will update this post with further information as it becomes available.

Update May 9, 8:30am PST: There seems to be DDoS activitiy which is causing downtime for a small subset of our customers. Our network engineers and system administration team are working to block the attacks at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.

This entry was posted on Thursday, May 9th, 2013 at 8:09 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

16 Responses to “Connectivity Issues with US-West Data Center (Los Angeles, CA) (RESOLVED)”

Our site is down, hosted on Walsh I believe.

Our site is down, hosted on towns — towns is up but very very high cpu load (load average over 1000), seemingly timing out trying to talk to db server maybe?

“Vienna” look down

burnet is very slow (pages take > 1 min to load)

May 9th, 2013 at 8:40 amDarla Dimple Says:

“d” as in “d”dos?

May 9th, 2013 at 8:44 amour biggest event of the year Says:

Down on our busiest day of the year. We will have to move after this.

My site is not working as well. I dont’t mind my personal site but I have few other client sites and a couple sites went for client review. This is not the first time Dreamhost got attacked. I’m not happy, man.

@Madara: Your web server is not hosted in our Los Angeles datacenter, it’s in our Virginia DC. I’m having someone look into the issue for you, but please post only if your web server is actually in our Los Angeles DC. You can check which datacenter you utilize in the control panel.

How about cloudflare?

Our site is down, and has been since 8am. I cannot log in to find out where our host data center is located because our login password no longer works (is no longer active???) and we were advised by DreamHost that we need a signed letter from our Board of Directors, on letterhead, with email addresses listed before we can access or update our password. While I understand the need for security measures, this puts us at a significant disadvantage as the site is down, and we cannot log in to find out any more information. If someone could check our host, that would be a great help. Thank you.

Our site is down as well as of 1450 PST. Any update?

Our website is still down as of 7:30pm PST. I cannot log into our control panel… same situation as Tess has. Can you please confirm whether this is related to today’s outage? Thanks.

You can ignore my previous post. Turns out there was a separate problem with our site.

Our forum is still down, unable to connect.

lovevsmodels.com/forum

All my site that are on the West Cost Data center are STILL down. So, I would say that the problem is not resolved.

All my sites are down!!!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency network maintenance in our US-West Data Center (Los Angeles, CA) (RESOLVED)
10:47 am

Emergency network maintenance in our US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 6th, 2013 at 10:47 am PST) by andrea

We are starting an emergency network maintenance shortly, around 11am PDT, in our US-West Data Center (Los Angeles, CA). This will affect mostly mysql servers, and a very small subset of web servers. No mail services will be affected. We are not expecting any customer-facing issues, but our teams are closely monitoring this maintenance nonetheless.

As always, please contact support if you have any questions. You can check if you may be affected by this in your DreamHost panel.
We will update this status post as the maintenance moves forward and monitor comments posted here.

Update May 6th 11:00am PDT: This maintenance is complete and we are monitoring to make sure that no customer services are affected.

Update May 6th 11:20am PDT: We have monitored our services and found no issues related to the maintenance. If you see any issues that you believe may be related, please contact support.

This entry was posted on Monday, May 6th, 2013 at 10:47 am and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Secure certificate changes for IMAP and POP on homiemail-master and homiemail-sub3 email clusters (RESOLVED)
2:55 pm

Secure certificate changes for IMAP and POP on homiemail-master and homiemail-sub3 email clusters (RESOLVED)

Posted (May 2nd, 2013 at 2:55 pm PST) by andrea

We are happy to report that your SSL connection to IMAP and POP is now using a non self-signed secure certificate. If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, you will likely run into a warning when you log into your email client again, stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that.

For now, this is only affecting IMAP and POP access for customers on the homiemail-master and homiemail-sub3 email clusters. To check your mail cluster, log into your DreamHost panel and click your Account Status on the upper right hand side.
We will post again when we roll this change out to the other email clusters, and when SMTP is ready to go as well.

This is good news for those trying to use gmail as POP or IMAP to access their DreamHost email, as they can now use that service without warnings and problems.

Sorry about the scare, and please contact support if you have any questions.

Update May 16th 10:15am PDT: The changes have now been applied to all of our mail servers. If you have any issues connecting to your email account within your email client or sending/receiving please make sure you make an exception for our new certificate. Once you have done so, you should have no issues connecting or sending/receiving. In some cases you may need to delete our old certificate in your email client and accept our new one and you should be all set.  If you have any questions or concerns, please contact support, so we can assist you.

Update May 3rd 2:50pm PDT: Due to unforeseen complications, we had to roll back the SMTP certificate. You may have seen email sending errors, but those should be gone now. We’ll attempt to roll out the new ones again, but we’ll make sure to update this post to give you the heads up about the progress. Apologies for the inconvenience this has caused.

Update May 3rd 12:16pm PDT: We are going to change the SMTP secure certificate on your email clusters within the hour. As a reminder, there is nothing that needs to be done on your end, this is just a heads up that the change is coming and is initiated on our end (ie. safe). The other email clusters are in line for the change, but we’ll announce it in a separate post when it’s about to happen.

This entry was posted on Thursday, May 2nd, 2013 at 2:55 pm and is filed under Announcement, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost panel connection issues (RESOLVED)
12:56 pm

DreamHost panel connection issues (RESOLVED)

Posted (May 2nd, 2013 at 12:56 pm PST) by andrea

We are currently experiencing connection issues to the DreamHost panel. It’s being investigated, and we’re expecting it to be online soon. Please note, this is also affecting your mailbox manager at mailboxes.yourdomainnamehere.com . We’ll post updates as this progresses. We apologize for the inconvenience this is causing.

Update May 2nd 1:06pm PDT: We’re happy to report that we restored connectivity manually, and the DreamHost panel and your mailbox manager is now back online again. We appreciate your patience. If you have any remaining issues, they are likely unrelated, so please contact support with the details, so we can assists you.

This entry was posted on Thursday, May 2nd, 2013 at 12:56 pm and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “DreamHost panel connection issues (RESOLVED)”

May 2nd, 2013 at 1:03 pmNick Fotopoulos Says:

Back online now…

This is also affecting all Announce List processes. If you’ve got a subscribe/unsub form on your site that relies on the API it will produce a page not found when submitted, and if you’ve just added 5k email addresses to an Announce List (like we just did) this morning, none of the confirmation links in the emails that were sent out will work and you will start to receive hundreds of complaints from people trying to complete their subscriptions. Murphy’s Law timing…

Phew! Back online now. Hopefully it will stay!

I’m still having problems receiving email. Not online here.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking issues in our Los Angeles Datacenter (RESOLVED)
5:39 pm

Networking issues in our Los Angeles Datacenter (RESOLVED)

Posted (May 1st, 2013 at 5:39 pm PST) by Seohee

We’re experiencing a network connectivity issue in our Los Angeles Datacenter. This will affect your sites, MySQL, email, webmail, the DreamHost panel, and Dreamhost.com. We’re currently investigating the cause and working on returning all services. We’ll provide an update shortly. You can check if you’re affected by this outage by looking up your data center location on your DreamHost panel.

We apologize for the trouble and appreciate your patience.

Update May 1st, 11:35PM PDT: We are confident that we have identified and resolved the hardware that caused this issue to occur. Thank you for you patience and communication during this incident.

Update May 1st, 6:34PM PDT: Services remain up and we’re continuing to monitor them to ensure stability. We’ll resolve the issue once we’ve confirmed we’re in the clear. We appreciate your patience.

Update May 1st, 6:19PM PDT: The networking issue that caused the downtime has been cleared up and services are returning. However, we’re still investigating the root cause, and are holding on resolving the matter once we’ve gotten confirmation we’re in the clear.

Update May 1st, 5:59PM PDT: We’re still investigating the issue and are working on getting our services up and stable. For now, you will continue to experience some intermittent downtime until we’ve made further progress. Again, we apologize for the inconvenience and appreciate your continued patience.

This entry was posted on Wednesday, May 1st, 2013 at 5:39 pm and is filed under General Outages, Major Outage, Multiple Server Issue, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

39 Responses to “Networking issues in our Los Angeles Datacenter (RESOLVED)”

The panel is down.

My mysql server has been unreachable for 20 minutes

My website is also down.

Recent downtime is getting frustrating.

“Recent downtime is getting frustrating.”

temporary? mysql not working….

Recent downtime is getting extremely frustrating…

why do we keep putting up with this?

This is so frustrating. This time that you were down was extremely important to our company. You just lost us a lot of business.

Nice – I noticed, but hard to report when panel is down.

suggestions for reliable service?

Here in Brazil, returned to normal

When will the panel come back up?

“why do we keep putting up with this?”

why can’t they just come clean with what has been going on the past few months and how they will fix it. has anyone been fired, any equipment replaced? people wouldn’t be so upset if they felt like something was being done and the issues were out in the open

panel, email and website – all down :(

panel is pingable with intermittent timeouts

We came back for a couple of minutes. Now we’re down again. Whats up?

down here… on a busy traffic day too #fml

#fml

How does this status blog stay up when the panel and dreamhost.com goes down? Do you guys use another hosting company for this site?

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

Huh? So what’s the point of this? So that we can talk amongst ourselves when DH has problems? Why do they ask for our email address?

wow, I was actually dumping dreamhost while this happened. Thanks, actual big boy server, for not going down. My email is down, but my sites are up. Why? Because I initiated a transfer out of dreamhost at around noon.

No webmail.

Still waiting.

@Chris – While we do our best to monitor the comments and assist you as best as we can, there is no guarantee that we are monitoring them 100% of the time.

This happens too often ! I am really frustrated and disapointed !!!! will not stay your customer longer

@Charley – Our Los Angeles Datacenter is currently experiencing the issue, and Dreamhoststatus.com is not setup at that location.

Email down.

Thanks Seohee – v. good to see that you do monitor. Thanks.

Website and panel are loading, but extremely slowly.

Sweet… My paid marketing campaign that is running and can’t be paused is really earning its keep, sending traffic to a dead landing page. I used to buy into DH and brought tons of sites over (sent some affiliate referrals, too). Now, I’m migrating everything away from DH because of consistent problems and a lack of interest about the problems by the staff. Another example of how DreamHost leaves you out on your own, even when the problem is obviously their fault.

http://www.whoishostingthis.com/dreamhoststatus.com – look at it yourself. Not hosted at Dreamhost :P

Still no luck with mysql database access on my sites

http://imm.io/14FA6

seeing lots of 500 Internal Server Errors (see picture)

@Bryan: that is common practice for hosting status pages for all hosting companies. How do you expect Dream Host to give us updates if they are having issues with their services?

Linode’s status page is hosted by another company as well.

I’ve still got nothing.

Site is down. Wasn’t notified. Not happy.

It (the site) was back up couple of hours ago and it went down again and not accessible for past 15 minutes. Did something break again?

Thanks

this is unacceptable i work in the music industry and constantly have important emails coming in to my email you guys need to get your shit together and stop making excuses

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Thursday May 2nd 6:00pm PDT (9 hours, EXTENDED, RESOLVED)
4:26 pm

Network Maintenance US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Thursday May 2nd 6:00pm PDT (9 hours, EXTENDED, RESOLVED)

Posted (May 1st, 2013 at 4:26 pm PST) by Ralph

On Thursday, May 2nd at 6pm PDT, we will be performing a core router software upgrade in our US-West Data Center (Los Angeles, CA).
We are scheduling a 6-hour maintenance window to allow a safe pace to test each step and monitor the successful completion. During this upgrade, the network traffic will temporarily flow through routers that have been thoroughly pre-tested, to minimize downtime. This is a routine maintenance that we have been able to accomplish in the past with seconds of impact on service.
As always, please contact support if you have any questions. You can check if you may be affected by this in your DreamHost panel.
We will update this status post as the maintenance moves forward and monitor comments posted here.

Update Thursday, May 3nd, 3:36am PDT:As of now, the maintenance window is now over, and the move to new hardware is complete! We thank you for your patience, and we apologize for having to extend beyond our original estimated completion time!

Update Thursday, May 3nd, 3:15am PDT:We’re past our window, however, we’re _almost_ done! We anticipate less than 30 minutes to complete

Update Thursday, May 3nd, 2:50am PDT:We’re back and in good shape!

Update Thursday, May 3nd, 2:48am PDT: We’re making a final change here, all of our Los Angeles (Garland) datacenter will be offline for about 20-30 seconds.

Update Thursday, May 3nd, 2:34am PDT: Central services are backup now. Panel, and webmail should be functioning normally.

Update Thursday, May 3nd, 2:30am PDT:We’re seeing problems on our web panel, (https://panel.dreamhost.com) and web mail (http://webmail.dreamhost.com). We’re investigating this and hope to have it backup ASAP.

Update Thursday, May 3nd, 2:20am PDT:We’re moving networks over to the new hardware, one at a time, checking each as we go. You may notice a few seconds of downtime while this happens, but things should recover shortly after.

Update Thursday, May 3nd, 2:15am PDT: Central services are up and running on the new hardware, so we’re moving right along to the next network subset.

Update Thursday, May 3nd, 2:09am PDT:We’re moving networks now, most notably, our centralized services. This means our web panel, webmail and dreamhost.com could be intermittently available.

Update Thursday, May 3nd, 1:30am PDT: Preparations are done, its time to start moving networks over to the new hardware. We’re almost ready to do that, and will update when we anticipate there might be downtime due to that.

Update Thursday, May 3nd, 1:09am PDT: We’re still working behind the scenes, and haven’t had any outside issues at all. The switch over to new hardware should happen soon. Unfortunately, we don’t anticipate being done by 2am PDT, so we need to extend the window one additional hour. We anticipate being done at 3am PDT, May 3rd. We apologize for this additional time being added.

Update Thursday, May 2nd, 11:20pm PDT: We’re still in a bit of a holding pattern at this point, making sure all related parts are properly configured for the switch over. At this time, all systems should be functioning normal. We’ll make the switch soon!

Update Thursday, May 2nd, 10:00pm PDT: As of now, we’re gathering up our data, and making sure we’re completely ready for the next step. The next step is to move to the core router upgrade. This includes wiping the router software and re-initializing and installing. From there, we can move traffic back to it, and test all the links and overall network functionality

Update Thursday, May 2nd, 9:30pm PDT: We’ve assessed the remaining work we have, and think it’s a good idea to extend this maintenance window 2 extra hours. This means we’ll conclude at 2am, PDT on May 3rd. We apologize for any inconvenience this causes, we just want to make sure we’re in the clear here.

Update Thursday, May 2nd, 9:11pm PDT: We’re back online now and verifying all services are in working order. Your domains and service should be up as well.

Update Thursday, May 2nd, 9:07pm PDT: We are currently seeing services down. Our main site, web panel, webmail as well as customer sites are down. We are working to restore this ASAP.

Update Thursday, May 2nd, 8:49pm PDT: Service to http://dreamhost.com has been restored. We’re pausing to investigate.

Update Thursday, May 2nd, 8:43pm PDT: A route we just moved caused our main site, dreamhost.com to go down. No other services should be affected by this. Our web panel (https://panel.dreamhost.com) and webmail (http://webmail.dreamhost.com) are unaffected. We’re working to restore service to http://dreamhost.com

Update Thursday, May 2nd, 8:10pm PDT: Internal moves went well, now we’re moving on to another subset of IP addresses. This time, the networks we’re moving can cause you brief periods of downtime. We’re closely monitoring this to ensure it goes as smoothly as possible.

Update Thursday, May 2nd, 7:50pm PDT: We are currently moving internal services off of our main core router, which will take some time. As soon as we have more updates to report, we will do so.

Update Thursday, May 2nd, 7:02pm PDT: At this time, we are nearly done moving customer-facing VLANs one-by-one to our alternate hardware and seeing no ill-effect. Things are on schedule and maintenance is proceeding. Stay tuned for updates.

Update Thursday, May 2nd, 6:51pm PDT: We have figured out our IPv6 woes and are continuing with customer-facing VLAN moves at this time. There may be slight connection interruptions as the VLANs move, but normal operation should resume immediately after.

Update Thursday, May 2nd, 6:35pm PDT: We have begun moving customer-facing VLANs to our backup network hardware. The first VLAN has been moved, but we encountered some unexpected issues with IPv6, so we’ve reverted this vlan back to the original hardware and we’ll be making sure that IPv6 will be moved properly before proceeding.

Update Thursday, May 2nd, 6:16pm PDT: Maintenance has begun. We are currently moving a subset of IP addresses that are not customer facing across network hardware, so the potential for customer impact is not particularly present at this time. More updates to follow.

Update Thursday, May 2nd, 5:56pm PDT: We are currently gearing up to perform our maintenance. We’ll be updating this post throughout the maintenance period. Should you have any questions along the way, please contact support!

Update Thursday, May 2nd, 11:00am PDT: The upgrade we are going to perform tonight has been pushed up 2 hours to 6pm, instead of 8pm. The post and title have been updated to reflect this change.

This entry was posted on Wednesday, May 1st, 2013 at 4:26 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

8 Responses to “Network Maintenance US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Thursday May 2nd 6:00pm PDT (9 hours, EXTENDED, RESOLVED)”

My web sites have just gone down with a “Connection timed out”, and I cant ssh in to haddock. Webmail and popmail seems OK though.

ERR_CONNECTION_TIMED_OUT to panel.dreamhost.com

May 2nd, 2013 at 9:06 pmPhil Again Says:

If it helps at all, my traceroutes are getting here …

15 te7-1.mag01.lax01.atlas.cogentco.com (154.54.28.142) 229.920 ms 229.348 ms *

… but no further.

May 2nd, 2013 at 9:11 pmPhil Again Says:

All good again!

Good luck guys. Crosses fingers

May 2nd, 2013 at 9:16 pmDH Brian A Says:

@Phil, Gabe, SV:

Thanks for letting us know. We did experience a connection interruption of about 5-10 minutes to our Garland Datacenter, but our network team has solved that problem. We are sorry for any inconvenience caused.

All of the WordPress sites I have hosted have gone down with an error (Parse error: syntax error, unexpected T_VARIABLE in /home/xxxx/xxxx/wp-includes/functions.php on line 192) since this maintenance.

@Tyler:

your WordPress issues are very unlikely related to this problem. I’m replying to your support request right now.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Ongoing mysql hardware maintenance in Virginia (US-East) Data Center
12:18 pm

Ongoing mysql hardware maintenance in Virginia (US-East) Data Center

Posted (May 1st, 2013 at 12:18 pm PST) by andrea

We are conducting mysql hardware maintenance in our Virginia (US-East) Data Center. This involves rebooting the servers, one at a time. This means about a 30 second downtime, and our data center operations team is closely monitoring the reboots to avoid any issues.

You can check if you may be affected by this in your DreamHost panel. We appreciate your patience while this maintenance is completed and apologize for all inconvenience caused. If you have any questions or concerns, please contact support, so we can assist you.

Update May 23rd 12:00 PM PDT: We have completed the needed changes and reboots for this maintenance. The servers affected will now have better connectivity and will be better able to be managed and monitored. Thank you for your patience.

Update May 18th 7:00 PM PDT: We are still working through the list of machines, so this is ongoing. The maintenance has still not caused any service issues, and we anticipate it will be done within a few days. Thank you for your patience!

Update May 6th 2:50pm PDT: Just a quick update that there is still some mysql servers remain to be worked on, but so far the maintenance has proceeded without causing service issues. We appreciate your patience while we complete the work, likely by the end of this week.

Update May 2nd 11:25am PDT: We’re taking a quick break today from the reboots then continue on tomorrow and early next week. There is a lot of machines to go through, and we’re keeping a very close eye on every single reboot to avoid any unnecessary downtime in your mysql service. So far so good, but if you have any questions, please don’t hesitate to contact support.

This entry was posted on Wednesday, May 1st, 2013 at 12:18 pm and is filed under Announcement, Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency hardware maintenance in Virginia (US-East) Data Center – Tuesday, April 30th 8pm PDT (RESOLVED)
4:33 pm

Emergency hardware maintenance in Virginia (US-East) Data Center – Tuesday, April 30th 8pm PDT (RESOLVED)

Posted (April 30th, 2013 at 4:33 pm PST) by andrea

Please accept our apologies for the late notice – we’re conducting emergency maintenance on the DOS attack mitigation hardware located in our Virginia (US-East) Data Center tonight, starting at 8pm PDT.
This is necessary due to the issues we experienced earlier today. We have been working with the vendor to diagnose the root cause, and will continue doing so later (at a less disruptive time of day for the majority of our users).

We are expecting this maintenance to be non-customer facing, but we are standing by to cover all possible situations. We appreciate your patience while we’re working on resolving this and apologize for any inconvenience this may be causing.

UPDATE: 6:28 PM PDT: We are resolving this issue as the DDoS mitigation hardware has been reported by our networking team as functioning correctly.  Please contact us if you still have any questions or concerns.

UPDATE: 5:58 PM PDT: We have all systems go, everything is turned on, and we are not seeing any issues. We will continue to monitor to make sure everything stays working without any possible problems.

UPDATE: 5:00 PM PDT: We are moving up this maintenance by three hours, so it will begin at 5:00 PM PDT. Again, it should not be customer facing, but we are prepared for anything, so please contact us if anything gives you problems!

 

This entry was posted on Tuesday, April 30th, 2013 at 4:33 pm and is filed under Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Emergency hardware maintenance in Virginia (US-East) Data Center – Tuesday, April 30th 8pm PDT (RESOLVED)”

Is this also affecting outbound traffic on brussels in US – West? PHP curl is taking a long time. Waiting for support to respond.

@JeanCarl, this is not affecting your server. Our support representatives will be in touch with you regarding your issue as soon as we can.

All of my dreamhost mail is down (via IMAP), webmail is working, albeit slowly…

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking issues at US-East Data Center – RESOLVED
3:37 pm

Networking issues at US-East Data Center – RESOLVED

Posted (April 30th, 2013 at 3:37 pm PST) by Elizabeth

Our admins are working on an issue with our East Coast datacenter. According to their reports, one machine is blocking access to a section of the datacenter.  At this time we don’t have an ETA, but we will update this post as soon as possible.

UPDATE April 30th, 4:10 PM PST: The issue was related to the earlier outage. The problems are now resolved, and you should no longer see any issues. If you are still seeing problems with your hosting, please contact support. Thanks you for your cooperation and patience while we worked on this issue.

UPDATE April 30th, 3:50 PM PST: It looks like this might be related to the outage we were working on earlier today. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

This entry was posted on Tuesday, April 30th, 2013 at 3:37 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Networking issues at US-East Data Center – RESOLVED”

April 30th, 2013 at 3:40 pmMichael Hofer Says:

“one machine is blocking access”

Hmmm. Occupy DreamHost East raises the bid… please don’t mace the servers.

Going on 7 hours now where my site has had almost no accessibility.
Very frustrated. Client is not pleased.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity Issues with Virginia (US-East) Data Center (RESOLVED)
9:12 am

Connectivity Issues with Virginia (US-East) Data Center (RESOLVED)

Posted (April 30th, 2013 at 9:12 am PST) by Justin@DH

We are currently seeing issues to servers in our Virginia data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

UPDATE April 30th, 10:50AM PST – The cause of the network issues has been identified and is now resolved. We are continuing our investigation into what triggered this and will plan on how to prevent it happening in the future. If you are still experiencing issues with your hosting, please contact support. Thanks you for your cooperation and patience while we worked on this issue.

UPDATE April 30th, 10:30AM PST – We believe we have found the cause of the issue and are going through our monitoring system and resolving any remaining issues that may be remaining. We hope to have all services restored and back to normal by the next update. Thank you for your patience while we finish resolving this issue.

UPDATE April 30th, 10:10AM PST – The investigation is continuing and we have made some progress, with some services restored at this time. We are continuing our work on investigating and fixing the remaining dns resolution and network routing issues. We are working hard to get your service restored as soon as possible. Thank you for your patience.

UPDATE April 30th, 9:50AM PST – We are continuing the investigation into the cause of the network issues. We have made some progress, and some services has been restored, but there still continues to be dns resolution and remaining network routing issues. We are working hard to get your service restored as soon as possible. Thank you for your patience.

UPDATE April 30th, 9:30AM PST – We have our entire team on the case and we are still investigating. We will update again soon if we have any news or if the investigation is continuing. Our apologies for the downtime caused by this network issue.

This entry was posted on Tuesday, April 30th, 2013 at 9:12 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

72 Responses to “Connectivity Issues with Virginia (US-East) Data Center (RESOLVED)”

MYSQL is DOWN!!!

MySQL, FTP, HTTP – everything is down. Thanks for working on it!

Most of the time I’m not affected by these outages… Not this time… Please fix it ASAP.

Well at least I know my new account is hosted in Virginia.

@JJ & @Kimberly & @Arturo: If you are in the Virginia datacenter you would be affected. We have our entire team on the issue and will have it fixed as soon as possible.

How can you tell what Data Center your service is on – or is Virginia the only one?

Site is down…. thanks for working to fix it…

Odd.. I’m still Shell’ed into my VPS – but I can’t restart the machine or ping anything else by hostname? Is it DNS issues?

Well that took a while, chartbeat was telling me the website was down half an hour before this went up. I’ve been frantically trying to find out why and finally this pops up… what was the delay?

This is effecting our business… Please hurry! In Denver Here…

April 30th, 2013 at 9:24 amGustavo Malamud Says:

How can I know if my server has been affected?

However: pinging by IP address seems to work fine. i.e. ping my work network from my VPS, works fine. By hostname, it fails.

April 30th, 2013 at 9:25 amCatalunya is not Spain Says:

All my webs are down!!! Mysql webs and html webs!!!

When will do you think that this problem will be solved? I am losing more than 6.000 USD/hour

I hope you can solve the problem soon.

Thanks

Thank you for working on this.
We notices an issue with the coockies (spelled correctly) exploit over the last few days. Might be part of your issue.

Server is down. Mysql is ok (external connectio)

Thanks and Go Go Go.

@James L: Thanks for the info. This seems to be a network hardware issue but we will keep an eye open for other possibilities.

Many thanks. That was quick – I’m back up and running.

It appears that about half our sites are working as usual, but the rest aren’t. All on same server though – which is interesting. Good luck DH support – we’re rooting for ya!

MySQL says: #1040 – Too many connections
this is prety sure a hacker attack to the databases

Some do / some don’t on the same server + my VPS can’t resolve hostnames (but IP addresses work fine) tells me DNS issues.

I can ping the internal DNS servers, but nslookup fails on all domain hostnames.

root@ps103367:/home/fnsnetps# cat /etc/resolv.conf
search dreamhost.com dreamhostps.com
nameserver 208.113.157.202
nameserver 208.113.157.201
nameserver 208.113.157.203

root@ps103367:/home/fnsnetps# nslookup yahoo.com
;; connection timed out; no servers could be reached

@Magno: We are still looking into the issue, we are not 100% back up, but I’ll check out your mysql if there is still an issue.

Google DNS from my VPS has no problem doing an NSLOOKUP:

root@ps103367:/home/fnsnetps# nslookup yahoo.com 8.8.8.8
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
Name: yahoo.com
Address: 98.139.183.24
Name: yahoo.com
Address: 98.138.253.109
Name: yahoo.com
Address: 206.190.36.45

@justin I’m still down with the same too many connections message as @magno

We’re still having issues with our MYSQL here as well.

I switched my VPS to use Google’s DNS servers for now until they get this fixed. For now – I’m online. Perhaps Justin will give me a heads up when it’s fixed about whether I was on-point or not.

Matt

@justin I’m still down. both ftp and mysql.

Please as @Catalunya is not Spain Says I’m also having serious business problems if sites are down.

Thank for your effort. Keep on fixing. :)

@Magno & @Matthew Glennon: Thanks for the extra info, dns resolution does still appear to be an issue and is being looked into. This would cause the too many connections for mysql and the nslookup issues reported.

April 30th, 2013 at 9:50 amCatalunya is not Spain Says:

Problem solved! Now my webs run ok, thanks a lot to the admins for their quick response.

I have a big business earning 3 million of USD/day and I cant have down my webs.

I’m still down as well, ssh and http.
Thanks for working this.

@Catalunya: Glad to hear your services are back up. We are not completely resolved with this issue, but hope to be soon.

@Paul: Sorry you are affected, we are doing all we can to get you back up and running.

Our sites are down.
Good luck!

For me, all my MySQL sites are down but plain old HTML sites are still up and rebooting MySQL in the DH panel returns status failed.

I’m still down. FTP seems to be okay.

We are still noticing that our website / sql are down. Appreciate any assistance with getting this back online asap.

our services are back up now – why did the servers not fail over to another data center to minimize the downtime?

@Dan Poynor: We are working on dns resolution issues which are affecting mysql, since they need to reverse dns an ip address to check if it matches %.dreamhost.com, etc. We hope to have it working again shortly.

Sites came back up for a bit and I was able to do some editing in wordpress but now getting 500 error

OUCH!!!!!
Time to load main page on website: 282 seconds
Time for a very simple MySQL query: 84 seconds
Time for app to download a small file via FTP: 29 seconds

site is down! Will you send mass email when server is fixed?

are there connectivity issues preventing my site from connecting to wordpress.com? dashboard connectivity from my site is spotty and can’t connect to update plugins, display stats, etc. i can manually open these sites myself, but connecting via my wordpress dashboard times out, unresolved host errors.

@Jay: Yes.. like we said earlier, DNS Lookups are failing from the datacenter.. so it wouldn’t be able to find wordpress.com.

Still all down… There is all down since 9:00 PST – One hour without services :(

Where are the DoS attack comming from?

@Yehudit: The Virginia datacenter has a larger amount of customers, so emailing all would take some time. If you want to get a response once the issue is resolved, including some details on what happened, please go ahead and submit a support ticket, and it will be answered when the outage is resolved.

@Jay: The dns resolution issues are likely a cause for that. Your site is located in the Virginia datacenter.

DH Panel says I’m hosted in LA, but all of my sites are down.

@Adon: Why do you think it’s a DoS? The services are there… I’ve been connected to my box this whole time – never D/c’ed the whole time. (I’ve been updating my box since before this started, installing 1.9.3-p392 of ruby on rails).

FTP service on john-henry-lee is affected. I can connect, but the server isn’t asking for a username. Yes, I’ve opened a ticket. :)

All my sites are online after changing the DNS Servers my VPS uses to lookup hostnames. (changed to Google). All functioning fine – very responsive. *not* a DoS.

I’m in a meeting trying to show a client my site and it’s down. This is EXTREMELY BAD!!!!!!

I’m loosing my client now!

@Roy Christopher: That’s odd, your site linked in your comment is hosted in Virginia for sure. Sorry you are affected by this issue.

@Adon & @Matthew Glennon: At this time there is no evidence of a DDoS causing the issue. We are still investigating though.

@Matthew I think is a DoS because web browser is saying (since more than one hour ago): server is too busy to answer requests – There are no services working since then…

what do you refer with “I’ve been connected to my box this whole time”?

Server does not respond to pings even. Argh!

Thank you for taking care of the stink. I do not envy your team. Both of my sites are down and is the first time since being hosted on DreamHost. It’s Unfortunate but I know that’s a risk of technology.

If you’re a *nix guru, and know your way around the bash shell – just ssh into your box, and change your DNS servers in your resolv.conf file – no reboot required – 9/10 of you will be online again. BE SURE TO COMMENT OUT the existing nameservers, not remove them, so you can go back when the crisis is over.

If you don’t know what I’m talking about or have to Google ANYTHING I just said, ignore me and wait it out.. you don’t have enough experience in the *nix world to perform that correctly and I’m not taking flack or responsibility for you being a novice.

My panel states that my mail servers are in LA (which I do still get mail), but I have database servers in Virginia with error as well for all of my sites.

Folks, take a breath. If your business is so fragile that some downtime threatens it then you should be spending a lot more money on making it fault-tolerant.

I can not imagine a universe where the dreamhost engineers are not working like their hair in on fire with an outage like this. Sure, downtime sucks for me too but I’m not going to panic over it.

Thanks for keeping us up to date.

Yeah very *nix guru but I don’t have a private server like you

@Adon: I-have-been-connected-ssh-the-whole-time.

When I changed the DNS settings on my box she came back up perfectly.
http://stats.pingdom.com/kzi9qrs1pynr/675475

Everything is still down. This is getting disturbing… >:((

Looks like I’m back up and running. Thanks DH!

@Matthew my-ssh-server-was-inaccessible-for-an-hour ssh and all!

Sites are ONLINE again. Thank you for your quick fix!
MySql seems working fine.

Great DH :)

April 30th, 2013 at 10:25 amnetrapal singh Says:

hello ,

we are on West US Datacentre CA but facing delay in mails of 15-20 minute. Are you sure only East US Datacentre has outage issue??

1) Were you trying to do this:
$ ssh your-domain-name.com ??

Because if you were – then it would fail – since you are trying to use DNS. You would need to use the server’s IP address instead.

2) Since you have a shared plan, it doesn’t make any difference anyway – you don’t have root and can’t edit the resolv.conf file.

Wait it out – you’ll survive. I’ve been with DH for 8 or so years now – I just recently moved to the VA Datacenter from LA – which has suffered everything from two-day long power outages to main network switch failures. We’ll come out the other side.

Daaaaaaaaaaaaeng. I be loosing BIZNASS!

I added
nameserver 8.8.8.8
nameserver 8.8.4.4

To the /etc/resolv.conf on my VPS and my vs sites are back up now. My shared sites are not through.

Perhaps dreamhost should consider using the Google dns as a fall back when theirs breaks?

I don’t like outages any more than anyone but I cannot help but scratch my head when I read the comments about people losing thousands of dollars and major clients because of this. If your websites are that critical to your business, why are you using “low cost” hosting?

The Dreamhost people work their butts off keeping this stuff (I’m not a geek this is all “stuff” to me) running. They are always helpful and courteous.

You have big money riding on this? Then spend what is commensurate with your business investment to keep it running 24/7/365.

Dreamhost people…keep up the good work. Web hosting is like everything else…what doesn’t kill you makes you stronger.

Blah. Just Blah.

@Shane: Good for you. I’m glad someone took my advice. Problem lies right there in nslookup.

@Bob H: Yep yep. I’m sure they are working hard to resolve the resolving ;) (see what I did there?).

Matt

@Justin & Matthew, thanks for the clarification! Hang in there :)

@Jay: No problem, but just so there is no confusion, I don’t work for DH.. I’m just a VPS customer that happens to be in the Virginia Datacenter and know my way around the Linux OS. I also keep a record of all of my assigned IP addresses for just this type of issue. Thanks for the props though.

Sites are up! That was FAST. Thank you!!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Dreamhost.com Down (RESOLVED)
8:26 pm

Dreamhost.com Down (RESOLVED)

Posted (April 25th, 2013 at 8:26 pm PST) by Seohee

Our main page http://dreamhost.com and webmail are currently down. We don’t have any details as of this time, but will make sure to provide you with another update shortly. We thank you for your patience.

Update Apr. 25th 9:11PM PST: We found that the root cause was due to a central database being rebooted. We are looking into why the reboot happened, and why it came back in a different state than its original to prevent this from arising again in the future. Again, we appreciate your patience while we got this worked out.

Update Apr. 25th 8:58PM PST: We have confirmed that ‘http://dreamhost.com’ and webmail services are back up and are confident that the issue is resolved. We are resolving the post. However, we will provide you with a final update once we have further details on the cause and any necessary preventable measures taken. We thank you for your patience.

Update Apr. 25th 8:39PM PST: The webmail and http://dreamhost.com are back up and running at this time. We are still investigating what caused the brief downtime for those services. We appreciate your continued patience.

This entry was posted on Thursday, April 25th, 2013 at 8:26 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

10 Responses to “Dreamhost.com Down (RESOLVED)”

now know how it feels.

:)

Not only dreamhost.com, all the webmail pages are unavailable. Gives following erroe:
Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Can’t connect to MySQL server on ‘10.50.0.5’ (4)

And this is why I’m leaving for another host, just waiting for them to approve domain transfer

Its back here!
i mean… online!

Considering my site must be functioning for Chicago Comic and Entertainment expo you could not have picked a worse time.

…yet the local providers who, while smaller in scale yet refuse to oversell remain online. Funny how that works. I was a customer years ago and don’t ever remember such frequent interruption of service. I didn’t come here for sales, I was looking for a VPS for QA purposes, but after reading all of these posts I may have to reconsider.

My site is currently unreachable at this time and I have many changes that must be made on schedule.

Starting to question this services up time.

I haven’t had many problems in the past but this is the second outage this month.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared email downtime on homiemail-sub5 (RESOLVED)
7:04 am

Shared email downtime on homiemail-sub5 (RESOLVED)

Posted (April 20th, 2013 at 7:04 am PST) by Oscar

We are currently experiencing a few issues with our homiemail-sub5 mail server which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Apr. 20th 8:00am PST: Our network engineering team has resolved the issue and all mail connectivity has been restored on homiemail-sub5. There was no email lost during this outage, but there will be a delay of one hour or less for all mail to be processed and delivered. Again, we sincerely apologize for any inconvenience that this caused and we appreciate your patience while we resolved this issue.

Update Apr. 20th 7:23am PST: Our network engineers are working to resolve this issue and will we continue to update this post as we have more information. We apologize for any inconvenience that this is causing you.

This entry was posted on Saturday, April 20th, 2013 at 7:04 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

45 Responses to “Shared email downtime on homiemail-sub5 (RESOLVED)”

everytime is homiemail-sub5. Problems with this mail server are a classical terror. Unfortunate its mine too :(

Thanks guys for fast announce. And sorry for my poor english.

Actually, I think this outage has been going on for a number of hours. I finally gave up waiting and reported the problem about 15 minutes ago. That may or may not have been the first DreamHost staff became aware of the problem but it’s been going on for a while.

Let me throw in that while I understand that things sometimes go wrong, it seems that email outages happen too often to be acceptable. I hope the DreamHost engineers are working on a permanent solution….a better backup system or whatever.

When you’re waiting for an important email and the system is down for hours it just isn’t acceptable.

webmail still seems to be working.
not an ideal solution, but workable.

I can sign into webmail here but messages are definitely not being delivered to my inbox.

It is ALWAYS homiemail-sub5 — third time in one week.

I’d be happy. to pay more for a stable mail server.

E-mail delivery has been sluggish for at least a couple days. Some mail sent from Hotmail and Gmail took several hours to be delivered to me. People would say “Didn’t you get my e-mail?” and I wouldn’t get it until a few hours later. Access was also unresponsive off and on last night. The last e-mail received here was at 1204z today (5:04 am PDT) And yes, I’ve stopped using procmail, or the ‘Rube Goldberg scheme’ as you call it, in favor of the cumbersome web panel mail filters which just don’t offer the same flexibility as a well-crafted procmail recipe.

Looks like the server is coming back to life. I can connect now and messages are slowing being delivered.

I’m going to invest in a flock of carrier pigeons; anyone want a piece of the action?

again, and again, and again.

For perspective: we detected mail delays on this system going back at least a WEEK. I have documented evidence that it was having a problem on Friday the 12th. So whatever it is it has been failing/attacked/whatever since then. Would be nice to know WHY this is happening (STILL).

We’ve already switched to a 3rd party for OUTGOING mail because of the blacklist problems on DH (AuthSMTP), I’m wondering about a 3rd party solution for incoming now.

time to move to gmail!

You guys are wrigte: this trouble seems to be a recurrent issue last two weeks.

Also I think dh has failed in provide info for tickets about this issue. I hope this has been finally resolved, users are now very disapointed with so many mail failures.

Also, I think it would be worthwhile to have the option to be pointed to a less crowded mail cluster, even with a cost for it.

This is beyond normal IT difficulties. homiemail-sub5 is clearly a lemon. Why won’t Dreamhost replace it? My clients are threatening to leave now if I don’t change hosts. I can’t defend Dreamhost any more. Their email has been down at least four times in six months. I have used about four hosts and this is by far the most unreliable email I have encounteret among them. homiemail-sub5 has to go!

Does anybody at Dreamhost actually read these things?

Still can’t send emails through php mailer …

I reported email issues onApril 15th and was told it was something with my settings. My settings have not been changed in any way for over 18 months. The email issue is still problem (on the DH side) and I know i am not receiving the volume of email that i normally receive per day. Please can we have a stable server… …this email issue is a major reflection on my business.

Delayed incoming AGAIN. Fourth time in a week and a half. Replace the server already.

Lots of incoming emails missing right now, on homiemail-sub5…

Yep, we are seeing delays again this morning too. I’ve got documentation of nearly an hour delay on some incoming messages.

People: compare the time stamp in the webmail “in-box” to the headers and when it arrives into a dreamhost address. For me, a message that arrived: Mon, 22 Apr 2013 07:18:41 -0700 (PDT) didn’t show up until nearly 8:14 PDT.

I am seeing delays again for the millionth time this month.

Just saw another delay. The same email, CC’d to two addresses on my domain. One arrived immediately, the other was an hour later.

Delays again for me : BCCd emails are not coming back to me. I submitted a support ticket

I’m experiencing > 20 min delays in receiving email this morning.

Yup – a 24 minute delay for a mail BCCd to myself just now

still delays in recieving mail. it has gotten worse around 11am MST. we have been experiencing delays since the junk mail filter server had issues. every day, around noon, users have a delay or can not recieve mail at all. hey there techs at dreamhost, is anyone updating this status?? its not a new issue. at least 5 days and i’m getting calls from staff every day about delays??? last year you upgraded your network but still having problems? as i understand it from a chat w/ your tech support, the server that the junk filter was on, was taken off line and the other servers couldn’t handle the load so the message qeue was backed up causing delays. so it looks like it is still happening!!!

Incoming mail still delayed today, Monday.

Indeed, mail is still delayed for me as well.

WTF Still delayed !!

Just got this reply
“Unfortunately we’re seeing some delays again today with emails going through the homiemail-sub5 email cluster. The good news is that we appear to have caught and correct the issue before it could become extremely large. The delays are getting better now as the mail servers work to catch up. Our admin team are working on another update to the kernel for these machines which they believe will help prevent this issue in the future. Again we’re very sorry about these issues. If you have any more questions please contact support through the Dreamhost panel.

I read this as “We don’t know what’s wrong but hope that software update and reboot will fix it” but perhaps I’m cynical…

I’d be a little more understanding of Dreamhost’s attempts to communicate with us if at the very least they didn’t rush to post “RESOLVED’ on this page over issues that are clearly not resolved. It just makes the whole Status Page idea seem like a joke. I am still experiencing >20min delays in both receiving and sending email.

got the same reply from a support ticket as xmd recieved. sure hope it works this time. i’m getting questions about why we don’t move to another host

Yep. Still seeing delays on some messages now. Getting lots of heat to move to a new host, and have found one. Depends on what DH does in the next 5 days.

Confirmed! Still have users reporting up 2hr delay…

in Poland actually mail server and panel is dead

Outgoing serwer for outlook is down – POLAND!

I noticed intermittent delays of incoming mail of 20-30 minutes on April 20th. I was seeing it again yesterday (Apr 22) at about 2:30pm, even though this status blog post said “RESOLVED” as of the 20th.

I filed a ticket and got the exact same response as ann and xmd above.

I also wrote in my ticket: “Please stop using http://status.dreamhost.com misleadingly”. I stated that this was my primary concern, so I could plan accordingly. They did not respond to that.

I agree 100% with Philip Owens above. “RESOLVED” means “FIXED”. If you post that the fix is underway [you believe], but *two days later* customers are still experiencing the problem, the lack of an accurate, non-“RESOLVED” update since Apr 20 is disrespectful as well as inaccurate.

p.s. Webmail worked for me the one time I tried it, but maybe the timing was coincidence. (It showed me a message that hadn’t come through via IMAP yet.)

p.p.s. I checked headers for delayed and non-delayed messages and noticed the timestamp on homie receiving the email was *earlier* than the timestamps on the sending servers (Google and two others). The timestamps were 20, 27, and 29 seconds earlier. I put that info in my ticket as well.

That is certainly surprising; is it relevant?

This is a nightmare. Delays on homiemail-sub5 AGAIN. Everyone put in your tickets so maybe it’ll get fixed for good this time.

Down again!

Down up down up. Chiming in that this is certainly NOT resolved at this time. Please correct.

There are definitely mail problems today. Also the DH panel is down.

ALSO NOTE: This post is not tagged with Email Issues, and thus those of us following the RSS feeds aren’t getting the notices.

I’m starting to think that the only emails going through are the ones with “test” in the subject line.

maybe dreamhost should do a kickstarter for a new mail server?

Our email is still down, what the hell Dreamhost. This is the 3rd time in a year.

Well, my mail is down on good old homiemail-sub5 yet again, and the web panel won’t load, either, so I can’t even contact support. It seems to me, as others have observed, that this particular server has more than its share of problems, and more than usual lately. Do I just have the bad luck to be hosted on the same server as one of your high-maintenence customers, or what? This is really starting to be bad for business and, personally, has caused me enough stress to give me an actual headache, again. I can’t keep doing this. Hope y’all can sort it out soon and keep it sorted. Thank you. :-)

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost Control Panel and Webmail Down (RESOLVED)
1:15 pm

DreamHost Control Panel and Webmail Down (RESOLVED)

Posted (April 17th, 2013 at 1:15 pm PST) by Justin@DH

The DreamHost panel (https://panel.dreamhost.com) and webmail are currently unavailable. Our administrators are working on finding the cause, so we’ll make sure to provide you with further updates shortly.

We are sorry for the trouble, and appreciate your patience at this time.

Update Apr. 17th 1:18pm PST: The issue appears to be networking related and has been resolved. Webmail and the panel are working normally again at this time, and we will continue to monitor the services for any further issues. We appreciate your patience while we got this worked out, and are sorry for the trouble this has caused.

This entry was posted on Wednesday, April 17th, 2013 at 1:15 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “DreamHost Control Panel and Webmail Down (RESOLVED)”

Webmail also seems to be down.

You are correct, webmail also appears affected. Updating the post to reflect that.

WEBMAIL DOWN!!!!!!

Just managed to generate this error while on the panel.

Go to domains, clicked edit on a domain …

Can’t call method “quote” on unblessed reference at /usr/local/lib/site_perl/Ndn/Common/Db.pm line 816.

Webmail back up for me. Panel too! Thanks guys!

Webmail back online, mail delivery working, however bounce message is being generated by the mail cluster on incoming mail

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues with some servers in Virginia (US-East) data center (RESOLVED)
7:32 pm

Connectivity issues with some servers in Virginia (US-East) data center (RESOLVED)

Posted (April 10th, 2013 at 7:32 pm PST) by Elizabeth

We are currently seeing issues to a limited number of servers in our Virginia data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it.

Update 7:46 PM PDT: It looks like connectivity is restored so we are resolving this incident. We unfortunately don’t have much information on what caused the temporary issue, but it was very brief. If you see any issues with your sites at this time,  please contact DreamHost support through your panel at Support > Contact Support. We appreciate your patience.

This entry was posted on Wednesday, April 10th, 2013 at 7:32 pm and is filed under Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Connectivity issues with some servers in Virginia (US-East) data center (RESOLVED)”

I’m having mysql connectivity issues, and no dh panel…

I JUST LOST A HAND FULL OF CLIENTS IN NORTH CAROLINA BECAUSE OF THIS CRAP!!!!!

The site is down for a few minutes, what´s up?

A pity the panel was one of the limited number of servers

The clock on this blog is out

Wow. So this is like downtime #20 over the course of March and April? Seriously?

At this point, you’re pretty much blatantly lying every time you claim to have a 99.9% uptime…

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail Delays on our Junk Mail System (RESOLVED)
8:32 pm

Mail Delays on our Junk Mail System (RESOLVED)

Posted (April 9th, 2013 at 8:32 pm PST) by Brian@dreamhost

We are currently experiencing mail delivery delays of up to a few hours affecting users who have our junk mail filtering enabled. Users without junk mail enabled are not affected.

The problems first started occurring this morning as a service stopped functioning on one of our junk filters. Further mails were successfully routed to other load balanced machines in the cluster, but one of them failed and was temporarily removed until load could be brought down. This unfortunately allowed the mail queues to grow rather large, resulting in the delays.

Our admins have determined the exact cause of the issue and will be looking into ways of further spreading out our mail load, as well as eliminating such single points of failure. The queue continues to diminish in size, so the situation is improving overall and delays are becoming less prevalent.

Update April 10th 12pm PDT: Our mail servers caught up last night, but unfortunately, we had a bit of a hiccup again this morning. It was much shorter in duration, and the mail delivery should be almost all caught up again. We are looking into procedures to avoid this situation again. Very sorry about the inconvenience this has caused.

This entry was posted on Tuesday, April 9th, 2013 at 8:32 pm and is filed under Multiple Server Issue, System-Wide Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

23 Responses to “Mail Delays on our Junk Mail System (RESOLVED)”

Hopefully the situation will be better once I wake up. FIVE HOUR delay here… Completely unusable for anything. This should have been posted earlier.

This is pretty ridiculous… getting client emails 6-12 hours delayed from this morning. I’ve disabled the junk filter on all of my domains as Gmail’s is a million times better, anyhow, but there should really have been some sort of announcement or posting earlier in the day.

ufffff no arribed mail!

Panel is down! All sites are down! Whats going on?

5 to 12 hour delays on emails from yesterday, after DH insisted there was no issue with email. Still experiencing delays. Please fix!!

My email continues to be delayed by several hours. Thanks for straining my client relationships once again, DreamHost!

we are in the middle of a storm and I,m in the rain business. People can’t CONTACT me !!!
this is really bad … 414-659-4499. somebody tell me whats going on please

We just made the switch to DH. I’m in the business of critical deadlines. The fact that you’re not keeping us updated is unbelievable. I’m still having issues and you’re costing me money and relationships. Update us hourly, then FIX IT!

this seems to be still going on!

We have received a few from yesterday morning, but still can’t get our current emails

Can DH please post an update on this? It’s clearly still happening for many.

This is absolutely ridiculous. Been a Dreamhost customer almost 9 years, and the last 2 years have been riddled with issues. Don’t know how much longer I can stay with this company that has frequent interruptions like this.

another few hours of a day lost all ready and no end in sight as far as I can tell

Still experiencing delays!

Still experiencing delays. Client relationships damaged :(

all my clients are upset!!!! when is this going to be over. it seems like there is always a problem with Dreamhost!!!!! how long will this last???? I’m loosing clients and files are being cancelled!!

Day 2 and our company email is STILL being effected. Incoming emails being delyed for hours! This is just another reason to dump dreamhost.

I’m at the end of my rope with Dreamhost email servers. Does anyone have a good alternative recommendation?

The same problem happened again with the Junk Mail disabled. I’m getting calls from my clients all day and lost some customers. Unbelievable that this is happening.

Junk mail seems to still come through great like it always has. Now I’m just waiting for my real emails.

I turned off our junk filter last night – emails started coming in. Now we’re experiencing delays on the mail servers again. I’ve stood behind dreamhost for 5+ years… I don’t know anymore….

still having problems. weird there is no new update about it

This problem is frustrating because there is little communication coming from Dreamhost. It would be beneficial to have notices come out on a regular basis. I am really questioning the technological stability of Dreamhost. Network issues are starting to become a regular occurance. How am I supposed to run a business with your service going down so often??

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connection issues in a segment of our network (US-West-Los Angeles) (RESOLVED)
2:19 pm

Connection issues in a segment of our network (US-West-Los Angeles) (RESOLVED)

Posted (April 9th, 2013 at 2:19 pm PST) by brians

Our System Engineers are noticing network problems on a specific row of machines in our Los Angeles,CA Data Center Facility. The affected machines are pasted below. We are diligently working towards resolving this issue. We will continue to provide updates as information is made available. Sorry for any inconvenience this has caused.

hoffa
batman
angel
biggs
dalitz
wolfsbane
karpis
provenzano
giancana
hornets
fitzpatrick
haddock
riddler
teak
vindemiatrix

UPDATE April 9th, 4:40 PM PST

Good news! All the machines are in relative good health, except for batman and riddler. We are closing this status post and communicate directly with customers on those two servers via email to keep them apprised of the situation.

If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.

UPDATE April 9th, 3:58PM PST

Services on the machines are close to being fully restored. Our System Engineers are continuing to fix a few non-responsive webservers. There are a few machine still reporting elevated loads. Once those issues have been fixed, we can consider this case closed. The exact cause of the outage is still being investigated as well. We sincerely apologize for any inconvenience this has caused and appreciate your patience. Please check back for updates.

UPDATE April 9th, 3:34PM PST

Services on the machines are close to being fully restored. Our System Engineers are currently fixing a few non-responsive webservers. Once those have been fixed, we can consider this case closed. The exact cause of the outage is still being investigated as well. We sincerely apologize for any inconvenience this has caused. Please check back for updates.

UPDATE April 9th, 3:07PM PST

The machines are back online. However, we are still noticing elevated loads on the machines. This will explain any lag that you are experiencing on your site. Our system engineers are still working towards having all machines operate at optimal performance. We will continue to provide updates as information is made available.

UPDATE April 9th, 2:41PM PST

We have noticed improvements across most of the machines. Our system engineers are still working towards restoring service on the machines that remain unresponsive. The cause of the outage is still being investigated as well. Sorry for any inconvenience this has caused. We will continue to provide updates as we reach a full resolution.

UPDATE April 9th, 2:25PM PST

Our Network Engineers continue to investigate this issue. Rest assured that we our exhausting all resources to restore service. Thanks again for your patience. Please check back for updates.

This entry was posted on Tuesday, April 9th, 2013 at 2:19 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

24 Responses to “Connection issues in a segment of our network (US-West-Los Angeles) (RESOLVED)”

Okay, this is about the 4th service interruption I’ve had in a month. I think it’s starting to get a bit unacceptable

From 6 hours ago was the first problema!! I sent you an support contact from 2 hours!! Please fix it!

my e mail has now not been working for 6 hours and no response from support about it. this really wasted a whole work day for me.

AGAIN?!? We should get a refund.

BACK ONLINE @ 2:55pm PST. Thank you Dreamhost!

Riddler has been flaky since about 4PM EST (1PM PST) … and database-powered websites still not working on Riddler (timeouts loading). Flat HTML seems to be fine.

I can now reach my site, but it is responding very, very slowly.

My site is still down…

@V We apologize for the problems that you have encountered. Please understand that we are really working towards minimizing downtime. Your continued patience is greatly appreciated.

@Mariano This outage started about 1H ago. You are likely not impacted by the outage. I will take a look at your issue and respond to your support request.

@Brian This outage did not impact email service. You are not impacted by the outage. I will take a look at your issue and respond to your support request.

The mails arrived 5 hours later my e-mail cluster is homiemail-sub4 (web cocobolo mysql accordion)

Actually flat HTML leading REALLY SLOW and database powered sites not loading at all .. on RIDDLER !

wolfsbane is back up, if a bit slow (wordpress sites at least)…this is the second outage I’ve had on wolfsbane within about 2 weeks. Not sure what sort of trouble you all are having in your infrastructure but I hope you get it sorted soon. I have been happy with you for nearly 5 years and would hate to see this trouble continue.

@Dave Mike Caitlin

The machines are back online. There is still noticeable slowness. Our admins are currently investigating. We will update once the issue has been resolved.

@LB I was unable to locate your account ( based on the information that you provide ). Please submit a support request through our Panel.

Hi guys

I got an email from Dreamhost, letting me know about the downtime in my shared machine.

I’ve had already noticed the down time (I have a ping system pinging my sites), but I didn’t contact you because I knew you were working on fixing the problem.

However, receiving an email acknowledging that you guys are solving the problem is a GREAT THING TO DO.

Please, continue sending this kind of email when there’s downtime in our machines. I know shit happens, but it’s good to know that you guys are working on it.

By the way, my sites are back already. I’m on hoffa.

Thanks

Sill can’t connect via ftp on Haddock

Server email cluster-master dosnt Works from 7 hours ago!!!

Seconding @Osiris on the effusive thanks for the email status update. I noticed the problem shortly before batman came back online, but only because I was attempting to figure out why I couldn’t connect to an unrelated machine. Having an explanation settles my mind a great deal.

mails continue to arrive late cluster is homiemail-sub4

aaaaand… my website is still down… on Haddock. :(

Riddler is still down. The status updates are not exactly what is happening …

Yup, batman’s still down. No SVN for me or my team.

Anyone know a good SVN host?

Seems to be on on going issue. Stability of email servers is like the plague for Dreamhost – this is the second time within a week email has gone down. If you want to maintain clients, you need to communicate better (6 hours after the fact is unacceptable).

I don’t understand. I am quickly losing confidence and am so unhappy with you right now.

For real? all day long? do you guys even know how much your down-times is costing your clients? I mean we pay for a dedicated server, a quite expensive monthly fee, and you guys haven’t even reply to our support post.

Our email is on homiemail-sub5 so, how much more time do you guys need to even say we’re working on it?

Damn!

You’re costing us MONEY.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web panel connectivity disruption (RESOLVED)
5:52 pm

Web panel connectivity disruption (RESOLVED)

Posted (April 6th, 2013 at 5:52 pm PST) by Elizabeth

We apologize for the issues you may be having with accessing your Customer Web Panel at this time. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible.

Please note, this should not be affecting any other services at this time (web, mysql, email, etc.) and is only causing issues when trying to access your panel.

Thanks for your patience!

Update: 6:50 PM PDT: After a brief outage, our panel services are up once again. Thanks again for your patience. Even though we were able to resolve this situation quickly, we are looking into what we can take away from this case to put even better checks and monitoring in place in the next couple of weeks.  If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.

Update: 6:31 PM PDT: The cause of the current panel issues is a problem with a critical backend data base server, which the panel needs to function correctly. We are diligently working on getting it running again, but until then, the panel will remain offline. Please note, this only affects the DreamHost panel, and all other services, such as domains, email or mysql should be functioning as expected.

Update: 6:10 PM PDT: Our admins are diligently working on the issue — we will post here as soon as we have more news!

Update: 6:00 PM PDT: We are still working on this issue — the machine in question is responding slowly. Thank you for your continued patience!

This entry was posted on Saturday, April 6th, 2013 at 5:52 pm and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

11 Responses to “Web panel connectivity disruption (RESOLVED)”

When will the panel be up and running?

We’re having problems with our email, but since the web panel isn’t accessible we can’t report it!

@Captain Bob: It will be back as soon as possible — from what I’m getting from the admins, that is supposed to be less than an hour, but I don’t have a firm ETA right now.

@Matthew: I’m so sorry! What problem are you having?

This is the 2nd big problem I had this year. Seriously not happy at all!

How would one setup a new domain on Dreamhost without using the web panel?

Every time this happens it is when I am in the middle of a series of tasks on the Panel.

At times like this I feel I should have a backup host so I can change the nameservers on my domain registar and point them at http://bit.ly/hostga

C’mon Dreamhost, a sense of URGENCY please

@losBones — Unfortunately, you can’t — I’m sorry!

We are getting errors on trying to ftp to our site (too many hosts connection login rejected). However, we do not have any hosts connected

Neither one of our two Mailman discussion lists seem to be working. Whatever gets sent is not showing up on the mail server. The last msg to make it to the server was on April 5 at 1415pdt.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues with Web Panel (RESOLVED)
6:47 am

Connectivity issues with Web Panel (RESOLVED)

Posted (April 5th, 2013 at 6:47 am PST) by chalise

We apologize for the issues you may be having with accessing your Customer Web Panel at this time. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible.

Please note, this should not be affecting any other services at this time (web, mysql, email, etc.) and is only causing issues when trying to access your panel.

Thanks for your patience!

Update 6:50am PDT: Thanks for waiting! One of the machines running the panel was having a few issues rebooting, which was causing the panel to not load. This has now been resolved, and the source of the issues located, and all panel services are back up once again. Thanks again for your patience. If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.

This entry was posted on Friday, April 5th, 2013 at 6:47 am and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Connectivity issues with Web Panel (RESOLVED)”

seems to be causing issues with my site? can’t access my site and it says database can’t connect.

para variar… no me extraña que algo asi pase

we got some email errors but seem OK now.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)
12:58 pm

Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (March 29th, 2013 at 12:58 pm PST) by Oscar

We are currently experiencing packet loss in our Los Angeles data center. This is due to a massive DDoS (the largest we’ve seen) that is being directed to our network. Our network engineering team is working alongside our providers to block the attack and we are making progress. We will update this post as soon as we have more information.

Update March 29th 1:40pm PDT: At this time we are no longer seeing any issues and have confirmed that the DDoS attack has been fully mitigated. We will continue to monitor our network for any future attacks that may happen, but we are now resolving this post as customer services are no longer being affected. Thank you for your patience while we resolved this issue.

Update March 29th 1:12pm PDT: We can confirm that the DDoS has been fully mitigated and is no longer affecting customer services. We are still closely monitoring the situation of course.

Update March 29th 1:09pm PDT: We are no longer seeing packet loss due to the DDoS attack and network functionality is back to normal.  We will continue to monitoring the situation along with our upstream providers and we are also looking for anomalous activity generating from within our network.

Update March 29th 12:57pm PDT: We have been in contact with our providers and are making progress in mitigating the DDoS. We are already seeing a significant drop in packet loss as well. We will continue to update this post as we progress.

This entry was posted on Friday, March 29th, 2013 at 12:58 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

49 Responses to “Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)”

Well that explains it. Have a lot of low volume domains hosted. One is a forum, however. Glad it is a long weekend.

that’s enough. I’m done. Good bye!

Having hard time accessing the admin page as well. Was able to get in to file a ticket, but other customers may not be able to access the page.

Good luck, please get it back up ASAP. I need to work on my site.

some issues in email services here (spain)

Yep, was about to file a ticket; my sites are nearly inaccessible and SSH connection to server is extremely slow. Good luck fixing.

How long are these problems supposed to last? Can we just get a window of the servers being up and down?

Too many days in a row with problems … this begins to be unacceptable.

I need someone on your Mgmt team to respond to my ticket. ASAP. You’ve had 3 critical issues that have taken my site down in 3 times 10 days….1 too many.

Thanks

Looks like its back up for me.

that was fast. you guys are awesome, but my clients are not awesome lol

Good luck guys! we’re expecting for your work

@Mitchell, Look up what a DDoS attack is. This is out of Dreamhost’s control for the most part. Anyone saying this is unacceptable doesn’t understand what is going on.

@Rafa: Our apologies for the downtime, but as Darren mentioned, this was an attack to our system. It’s been mitigated and your site is back up now.

I love reading all of these comments. “OMG the world is ending, I am losing Gazillions.” “I am leaving DH sucks.” I wonder how many of these people actually do anything. I bet less than 1%.

Anywho. DH I am impressed you nailed down a DDoS attack in less than 30 minutes.

spoke too soon

@Sara: I’ll be glad to address and questions or concerns you have. Please let me know your webid or acct number so I can locate your ticket.

@Lux: What’s your site? I’ll check it out right now.

http://www.esecurityplanet.com/network-security/how-to-prevent-dos-attacks.html

“Let’s start with the bad news: It is very difficult to defend against a sophisticated DDoS attack launched by a determined adversary.”

My question – is this directed specifically at DH, or at one of the sites hosted on it’s servers?

@xray: It was directed to a site hosted on our servers.

All my sites are down with database connection errors since the DDoS.

My sites on the Virginia servers and a ticket for no connection to mysql server has gone unanswered for hours. I can assure you, all my sites are still very down.

@Andrew: The issue with your sites is unrelated to the ddos. The mysql server for your databases was just rebooted. Your sites should be up in a few minutes.

@Nate: Most likely related to the same mysql server reboot as Andrew. Let me know one of your sites and I can look into it for you.

@Oscar – thanks!

Are we getting hit again? I was up and now I’m down.

@Oscar – check our http://www.lievt.org/ please, all sites requiring mysql still down

@Oscar – just came back up I think. Checking others.

Pretty impressive if DH was able to recover from a targeted DDoS in 30 minutes.

I am losing customers and the ability to run my business with the continued downtime from Dreamhost… anything but a dream.

@Dave V – Our apologies for the downtime. This was not caused by our network and was outside of our control. We were able to completely block the DDoS attack in a timely manner and your sites are up now.

Also, it’s “affecting.”

@Oscar – confirmed mysql connection good on all my sites now. This was unrelated to the DDoS I think, right?

@Nate – Yes. I was working on that particular server earlier today so I can verify it’s unrelated.

March 29th, 2013 at 1:41 pmconspiritech Says:

as far as I can tell no issues occurred with my sites. Paneland linked DHSoTMs were very slow for ~20minutes. I had assumed that was because a shared server host was being added alongside my VPS but perhaps not. remember to use simple and sanitary code, everyone!

This is too much, daily outages I’m switching hosts…

@Oscar – I guess I can thank you *and* blame you then, sir! Have a great weekend!

Is the SMPT issue sorted? I saw on the forums some others having issues..and this is since yesterday.

@Oscar: I also had mysql connection problems this afternoon. If you guys are restarting services leading to client-facing service downtime, shouldn’t you be letting us know first? Unplanned DDoS is one thing, but scheduled maintenance going unannounced is another, and I can’t find any recent mention of mysql server restarts anywhere on the status site. What happened there?

Still down here, is this unrelated?

Problems for about 75 minutes now…. site loading like they did back in the in the late 90’s on a 28.8k dial-up modem. What a Dream!

back up! Hurray :)

I honestly don’t give a damn what the cause of the downtime was… these issues have been going on intermittently for the last few weeks and its unacceptable. I pay a substantial amount of money per year to keep my online business running and I really dont need to keep worrying about one of the dreamhost servers going down and causing me to indirectly lose sales revenue. If my site is down , my clients will shop elsewhere … plain and simple

I will never host another site on Dreamhost again

i’m looking at Host Gator as a serious alternative

I agree that the intermittent issues have been occurring for some time & are just not acceptable. I typically continue to experience problems for some time after DreamHost posts that everything has returned to normal (like the current situation).

@Oscar – Ticket #5682547
Again today my VPS site was down while the other shared ones were up and running.

I’ve had your services for about two weeks and I’m beginning to think it was a mistake. Do you offer refunds?

Again??? Seems like every few hours I go to my site and get the nice big Cloud Flare your site is offline. I just completed a demo project for someone and I’m afraid to send them to the site because it will look bad if it is down.

@VIctor — Yes, you have 97 days to try out the service to see if you like it. If you cancel within that, the hosting fees are refunded.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » EMERGENCY MAINTENANCE: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 27th (RESOLVED)
7:43 pm

EMERGENCY MAINTENANCE: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 27th (RESOLVED)

Posted (March 26th, 2013 at 7:43 pm PST) by andrea

After some delays and postponements, we are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May 2012 is finally nearing completion. We started this project when we realized our network stability was in danger as we had outgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, March 27th, we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). We apologize for the late notice, but in light of yesterday evening’s incident, we feel it’s necessary to go ahead as soon as possible. This maintenance will start shortly and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail including Webmail, MySQL, the Web Panel and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update March 27th 9:00pm PDT: We have remained stable since our 4:10pm update and we are closing out this emergency maintenance. We will need to do a final maintenance which will be announced once we are ready to proceed with the final steps. As of now, the majority of our data center is on the upgraded hardware. Please be on the lookout for further maintenance announcements and thank you again for your continued patience.

Update March 27th 7:30pm PDT: We’re happy to report that the switch upgrades went smoothly, and are now complete. We are still not finished with the maintenance, but are nearing resolution. Please note that customer services are completely up at this time and not affected by this maintenance. We will provide you with an update on its progress shortly.

Update March 27th 5:20pm PDT: Apologies on the lack of updates. We have been in a holding pattern currently waiting on our hardware vendor’s assessment of the troubleshooting data we have passed them. We do have information on why we are having trouble moving all of our traffic to the new hardware. We are about to perform some further maintenance that will affect a small subset of our customers. This maintenance is a switch upgrade and we will be able to move it with minimal to no impact. We will update once that portion of the maintenance is complete.

Update March 27th 4:10pm PDT: All services are reporting live and functional at this time. Some of you may have noticed some connection issues recently, these are now resolved. One outstanding issue is keeping us from closing this status post. We thank you for your patience and will update you as soon as the last issue has been resolved.

Update March 27th 3:38pm PDT: We’re seeing a whole lot of improvement now. If you’re still seeing problems, please comment below, and let us know, so we can investigate individually.

Update March 27th 3:10pm PDT: As many of you in the comments have noticed, we’re seeing domains/IPs coming back online, but some service is still a bit spotty. We’re working on that, and should have more improvements coming!

Update March 27th 2:52pm PDT: We are seeing improvement in more and more areas of the problem networks, and are still working to get them all up and running. Again, we apologize for this, and appreciate everyone’s continued patience.

Update March 27th 2:35pm PDT: At this time we have seen some improvements in network performance, however we’re carefully progressing at this point, to ensure we don’t cause more problems with customers that are currently up. We will continue to update as we get more information.

Update March 27th 2:17pm PDT: We are still working on getting all services back online. At this time we’re still troubleshooting problematic networks and doing everything we can to get services back up as soon as possible. Again, we appreciate your patience on this.

Update March 27th 1:55pm PDT: At this time, services are looking pretty good and most things are online and running optimally. However, we are seeing some connectivity issues across certain IP ranges, and are working on getting those ranges responsive. We appreciate your continued patience while we’re working towards resolution.

Update March 27th 1:33pm PDT: Unfortunately, we’re still experiencing some networking issues in our US-West Data Center (Los Angeles, CA). We are investigating the cause and will post an update as soon as we have more information.

Update March 27th 1:11pm PDT: The maintenance is still going, but it appears that we ran into some networking problems. We are looking into it and working towards eliminating this, so we can proceed. We appreciate your continued patience.

Update March 27th 12:20pm PDT: All customer traffic is now routed through our new hardware. So far it is working great, but our teams continue to monitor and test to make sure it stays that way. We have some routes to move that do not directly affect your services. We will continue to provide you updates as we progress through the maintenance.

Update March 27th 11:57am PDT: First part of the migration is now complete. As you may have noticed, this caused some of the issues with our DreamHost web panel and DreamHost.com. As our next step, we’re moving to the customer machine rows now. We are expecting minimal service issues in this next step.

Update March 27th 11:41am PDT: This emergency maintenance is now in progress. We appreciate your patience while we’re working on this and we’ll be posting updates periodically.

This entry was posted on Tuesday, March 26th, 2013 at 7:43 pm and is filed under Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

154 Responses to “EMERGENCY MAINTENANCE: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 27th (RESOLVED)”

That’s cool. I was wondering why we were seemingly going up and down a bit. Does any of this have to do with the DDOS being reported by Spamhaus?

Is now in progress? It would have made more sense to work on this at night…

March 27th, 2013 at 11:54 amAndy Ringsmuth Says:

Been seeing mail go up and down the last hour as well (as of 1:48 CDT on March 27).

I have a feeling Dreamhost can’t handle the DDoS traffic that has been bringing down many many websites today and networks today.

That was the whole reason why they have been doing these upgrades to begin with if you follow the history of upgrades over the past 6 months. Hopefully this Emergency Maintenance resolves things.

@Bart: No. It has to do with a faulty router that caused the outages last night. We’ve bumped up this maintenance window to address this issue and prevent future outages of the same nature.

@Cyrus: We initially had it planned for tonight, but due to it causing further outages this morning we bumped up the maintenance window to address it.
@Mike: We have not seen anything to indicate this is DDoS related. As far as we can tell faulty hardware is to blame and we’re in the process of removing it now.

I thought this was set for 8pm PT. Why is it set for now? It’s the middle of the workday for some of us.

@Dawny: Cool. I was just wondering. Hope all goes well with the upgrade. I know how frustrating they can be. :)

Oh for fuck sake. Third time this week.

issues with email services here. web still working

Wasn’t this supposed to happen tonight LA-Time? This post said 8PM PST and it’s not even noon now!!

Seems like all my sites have been up and down all day; just as I send an email explaining why they’re down, they’re back. Then as I send an email saying they’re back, they’re down. Currently they’re down again.

It’s kind of a pain because our newsletter goes out on Wednesday mornings; it’s our busiest and most important day. I look forward to a flurry of emails from my clients and readers along the lines of “Y UR SITE NO WORK NOW!!! FIX IT!!!”

I’m always kind of amazed at the vitriol in the comments on these posts. You’d think webmasters would understand how annoying impatient, indignant comments from customers when something’s broken are.

Still, the server downtime is really becoming a liability for me. When this happens during the day it absolutely kills me – especially Wednesdays. I appreciate you guys doing everything you can to prevent it. I really like Dreamhost so I want to give you guys the business. Just know that this is really straining our relationship.

“we bumped up the maintenance window”

No. Actually, you had an unplanned outage, again, and decided to paper over it and call it maintenance. A maintenance window that starts NOW is not planned maintenance, it is an unscheduled outage.

Thank you for NO FUCKING NOTICE AT ALL. Your incompetence at running a service is rivaled only by your unprofessionalism at communication. Now go ahead, complain about my use of the word “Fuck”.

@Francesco & Damin: It was scheduled for later this evening, but as I mentioned above, these new outages this morning forced us to bump up the maintenance window to address these issues.

Wow guys, you’re killing the productivity of your clients! If its not my website, its webmail. Are you working on the maintenance now or is the current problems with webmail part of the problem you’re trying to resolve with the maintenance?

Ugh. Again, the fails during business hours are not making me, my partners, or clients happy. Granted, we keep weird hours, but still… Guess I’ll go do something else until, um, tomorrow? *bangs head on keyboard*

Does this mean my mail won’t be deathly slow anymore? Because it’s the slowest email server I deal with. Even the crappy personal account I have with GoDaddy isn’t this bad, and it is bad. I want to be happy with you, DreamHost. Really, I do. Please give me something to love!

It went up now its back up. Should I expect up and down for the next 4 hours?

I have a official launch of my website today at 6pm (GMT -3). My site goes down every minute because of this maintenance. His name is Saprissa.

Is there anything you can do?

*** went down

@Dmitriy: It is possible that that services will be up and down during the 4 hour window of this maintenance period. We’re doing our best to keep things up and running as much as possible during this time though.

I have a official launch of my website today at 6pm (GMT -3). My server goes down every minute because of this maintenance. His name is Saprissa. Is there anything you can do?

@ Carol: In total agreement wit you!
@ Dreamhost: hope you’re taking comments like Carol’s very seriously because it expresses a pervasive sentiment

@Vinicius: You haven’t given any information as to what domain is affected. If your site is in this datacenter, there isn’t anything we will be able to do until the maintenance is complete though.

Uh, you guys, unless you’re on a VPN and dealing with this BS why are you complaining?

I’ll bet that at least half of you (myself included) signed up for the service on a whim years ago due to how cheap the service is and how good the support was. And I also bet the majority of you haven’t had an issue for as long as you’ve been a customer.

So if you’re here complaining about “losing money, clients, etc.” why do you pay $10 a month for DH service? There’s premium services meant for businesses. That’s like if I catered my business meetings by going into 711 and buying up all the Tacquitos… You get what you pay for.

And all those “We’re going elsewhere” threats… Do you think they care? They basically give away hosting and domains on a monthly basis here. Acquiring new customers are quite possibly the least of their worries.

Have some patience and see what happens in a few hours. If you’ve still got issues, maybe then you do something about it.

The Spamhaus DDOS attacks may have nothing to do with this outage but that’s not what I’m going to tell my clients! ;-)

Server Offline Again ?

So, I’m trying to understand people. They complain that ‘stuff has been sucking’, so DH finally is trying to make things better, then people complain about downtime when the repairs are being made. If you are running just some ‘ho-hum’ website, then it really shouldn’t be a big deal. If you are running a ‘high-end’ website, then you should be fully aware of how these things happen and what it takes to correct them. And you call yourselves ‘professionals’?

Things break. Welcome to adulthood. Let them fix it, and we’ll all be happier. I have never yet had a vehicle, or anything else, break down when it was a ‘good time to have it happen’. Sheesh.

Sheesh all when you take in the amount of servers and drive space they deal with and offer at great prices, I would like to see any of you open your own damn biz and offer the same with a guarantee of no down time. S*&t happens and they are doing the best they can to keep it all up. Ya know for the worst whiners there is a good cheap option. Go buy a computer with a huge hard drive get a perm. ip addy and host your own damn service.

@Bart — right on, man! Couldn’t have said it better myself!!

I turned on the Cloudflare service that dreamhost provides as a configuration option – has kept my sites online through all these outages. wont work for email, but for keeping static versions of website’s up, it is sufficient

If only CloudFlare could be setup from Dreamhost for domain only websites….I suppose the hang up is because dreamhost wants to remain our DNS provider (which I suppose is necessary when the shared host needs to be replaced suddenly….or something.)

Bart is not the point.

All of us here know that problems happen … but you have to understand that we have customers … usually barely read an e-mail.

They will not know it happens and yes their website working …

Our complaint is that is happening almost every day and without notice …

Simply the site goes off the air …

If there is in my understanding a notice, we look days such time as the site server will be off alright, gives us time to advise customers and to put a banner on the site …

Do not just pull the cord and damned if customers

Seriously folks, I know this is when we all like to Internet Rage over Dreamhost, but read the heading. EMERGENCY MAINTENANCE. They made a business decision that not changing it now would have a greater impact on everyone than changing it later. And you know what? They know that this kind of response would happen one way or another, but think of it this way: Would you rather:

1.) Have the Datacenter Network Infrastructure fail again, producing a downtime that needs to be corrected, and then have another outage for the scheduled maintenance?
2.) Have an Emergency Maintenance that is shorter than the above mentioned failure, which corrects any foreseeable future issues?

I know the answer is “NO DOWNTIME”, but let’s be a bit reasonable. Does it suck? Hell yeah! Is it need. Definitely. Lesser of two evils here, people.

LOL@Leftovercake! Yeah, I never understood that…”I run a multimillion dollar web enterprise, and I am complaining when my $8.95/month server goes down.”

“Well, DH should be running redundant backups with multiple internet connections so nithing ever truly goes down!” Uhhh…yeah…for less than $10/month. And we wonder why people speak of the new “Entitled Generation”. “I WANT EVERYTHING, IT BETTER WORK CORRECTLY, AND IT NEEDS TO BE FREE/CHEAP!”

If you think it can be done free/cheap, with tons of redundancy, then please, oh PLEASE, go start a webfarm and let us know how it’s going. Until then, sit back, relax, and let DH do their work. And your company won’t collapse (and if it really is that important, you should be with a higher level of servers, anyway). I’ve been with DH for 10 years, been through many different server/network outages, and my companies are still doing just fine.

Damn. My car has been acting really weird lately. Just doesn’t run correctly. I’m going to take it to my mechanic to fix it right, and then complain the whole time that he has it, trying to get correct parts, get them installed properly, etc.

Does this suck. Well, yeah. But guess what?! It will be just fine. And again, if your website being online 100% of the time is of critical importance, then you probably shouldn’t be with DH anyway (no offense to DH).

Take a deep breath. Go for a bikeride. Go read a book. But for crying out loud, relax. “Oh, but they told us it would be 8pm tonight, and they started WAY early!” Well, more problems arose and they decided to tackle it now instead of waiting a few more hours, and having a few more hours of problems occurring before the problem truly gets fixed. Some of you people REALLY need to read some “How to live in the adult world” books.

I have customers, too. I’m keeping in contact with them. They’re ‘disappointed’ just as I am, but they are just fine with it. They understand. And if your business model doesn’t account for problems, whether it be ‘website downtime’ or ‘financial issues’, then I can suggest a few “How to correctly operate a business” books for you.

UPDATES?

I have sent a few notices to support that our home page was hacked! I havent heard anything back. Any advice?

@Mike: We’re still working on the maintenance, but for the most part services are up and running. We’ll post updates to the status post as soon as we have relevant news.
@Petros: I’ve forwarded your message on to our Abuse team to scan your site and send you a reply. You should hear from them soon.

My psmysql server is down although the host server homie-vsql73.dreamhost.com is accessible (pingable), my ps is not (pingable). Since Chat is down and it doesn’t make sense to create a ticket right now I am posting it here.

Hi, all websites/databases have been down for at least 1h.

Have submitted a ticket but no response.

When can I expect resolution? People are placing orders for Easter…

Please let me know!
Alex

you do know that not everyone pays $10 month..
I run a multi million dollar enterprise and I pay DH about $1500 per month as they have VPS and dedicated server.
and it has cost me a fortune the outage, but yes these things happen….. the first outage of a full business day was bad… but in 10 years it is the first time that has happened… the next outage was during business hours but under 2 hours of downtime… the current outage they warned us about…. so lets hope this is the last issue.

The DH control panels and user system is the best out there by far… to change users and migrate between vps,mysql, dedicated with the simple click of a button is not available anywhere els i have seen. I have been a DH customer for over 10 years.. and although it was the worst timing in history that full day of downtime was the first like that I have seen…..
Because dreamhost is offering chat support and their control panels are so good.. they get another chance in my book… lets hope this fix is the end to the outages…. and lets hope this fix is coplete within the 4 hours they have said.. good luck

March 27th, 2013 at 12:38 pmPatrick Monaghan Says:

Woo! Thanks guys. Not terribly painful and if it helps prevent the big pain in the future, I’m all for it.

Well, my site is fine (yay!), but I can’t access my email (boo). So all I’m concerned about is I can’t communicate with clients via email, and phone is not an alternative because I need a digital paper trail. Is this killing my business? No, but it is making us very frustrated here in the office. Our DH email service is always slow, but at least it’s usually there. Sometimes it drops a bit, sometimes it’s fine.

If I could afford better service, I’d get it, but we’re a tiny startup and it’s not in the budget this year. If DH’s improvements clear up the speed issues, we’ll be happy to stay. But when even my tiny business site and email service is clunky on a routine basis, I wonder what the best options are for us. I already know GoDaddy isn’t the solution.

@Jay: Your server is in the affected datacenter and we may still be working on it. If you give me more information I can get it looked into.
@Bakery: You didn’t give any info in your comment for us to be able to investigate your issue.

@Carol: We’re hoping these network improvements will fix the issues you’ve been having with your email, since most of our mail servers are in this datacenter that we’re upgrading network equipment in.

@Webb…if you have a multi-million dollar enterprise, and are paying DH $1500/month, you really should be operating your own servers with dedicated network admins. Then you would have full control and have a direct line to the people working on your ‘stuff’. Something to think about.

And if your multi-million dollar enterprise has server overhead of $1500/month, you are doing VERY well compared to the standard business world. Try operating a brick and mortar multi-million dollar business with $1500/month overhead. It’s not going to happen.

@Bart: I think I had come across another hosting provider that does that….for <$10/month they host your site in server clusters spread between two geographically separate data centers, so if the first location fails your site will automatically continue from the second location.

Though they weren't as flexible/feature rich as Dreamhost on what can be done on that package….

At least, I think dreamhost is better than my previous provider…their idea of daily backups was that they were running RAID1. Which they got away with unil they waited until both drives were failing before deciding that they should do something…and then they picked the wrong drive to replace first… Murphy strikes again. Silly thing was I waited until I lost my site a second time to finally jump ship.

Wonder if anybody ever asks for restores of their site to a previous time from their providers?

(We do get restore requests from time to time at work….but most users here make edits directly on the live website and don't keep their own backups… though I didn't start keeping regular backups of my own site until after the incident with my previous provider.)

danwy, yes I was a little vague. My psmysqlserver is psmysql2510 and one of the mysql databases that I have there is mysql.munzeemadness.com 208.97.133.221

Cuantas idioteces se leen aquí.

It is NOT working great! Some of my sites are down yet again.

Come on, this is getting really ridiculous. I’ve been waiting for like 3 hours to get some work done!!!

I’m concerned about the update that “All customer traffic is now routed through our new hardware” and “So far it is working great.” My sites are still down, including http://www.nslj.org, which is launching its spring issue today. Any more specific updates available?

“Update March 27th 12:20pm PDT: All customer traffic is now routed through our new hardware. So far it is working great”

Not so.
alfieland.com is still down!

All our website are off line. it seems SQL is not working. any news??

@Jay: Thanks for the info, we’re looking into it.
@Jen: I tested jaw-labs.com and it came up fine for me. Keep in mind, there may be periods of intermittent inaccessibility until we complete the maintenance and resolve this post.

danwy Says:
@Carol: We’re hoping these network improvements will fix the issues you’ve been having with your email, since most of our mail servers are in this datacenter that we’re upgrading network equipment in.

Great! Thanks.

I find it somewhat amusing/distressing how dependent we are on all of this. One big EMP in the right place and we’re all schmidt out of luck. A few outages are a pain, but not horrific (to me, at least). I’m off to go play outside until things calm down. Work was stressing me out anyway, even before the outage.

@ BART yes I am fortunate, I do have highly profitable company with low overhead. I have hired more expensive servers with dedicated system admins before and it has been no better.
I hire 5 full time developers including myself and I think its best to build our own redundancy… I did get caught out the other day with the full day of downtime etc.. but due to spreading my service over multiple locations right now I am still uo and working.. with only about 10% of my services down…. being with DH for over 10 years… and having good experience most of the time is hard to risk that i the hands of a new unknown company… and now that they have re activated chat support.. they have been quite helpful and quick.. when I have an issue and they have also made some changes to my srver for me to help speed things up..
5 years ago I got a new technology website I built on the front page of digg.com dreamhost didnt skip a beat with 3 million visitors in 2 days… I then thought I better upgrade and i got a dedicated server.. very expensive.. I made the front page again and it crashed.. and I got removed off front page of digg.. so I went back to dreamhost.

Their control panels and ease of use…. i have not seen anywhere els…

I have 30 staff that use a crm all day, thousands of customers delivering online services for… and im confident with Dreamhost with the $1500 per month.. after some strategic adjustments..

@Richard: This issue has not been going on for 3 hours, so it sounds like it may be unrelated. I did test your buildrr.com domain and it seems to be resolving with no issues.
@Alex: We’re looking into your site.
@Wysocki: We’re looking into your site as well.
@Giampietro: We need more info to be able to investigate.

@danwy,

yes my main domain is fine. but two others are not:
http://www.sagesociety.com/
http://www.leeonenessfoundation.com/

Can you look into our site as well: seatosummitworkshops.com ?

Seems to still be down.

Thanks.

@Webb…you just clarified the point I am making. If you had the *same* experience spending a whole lot more money and having on-site staff, then maybe DH isn’t so bad (not that YOU were complaining about them, but for the people that are). These are computers and electronics. Things are going to go wrong and break. It happens. Relax. They’ll fix it. :)

@ DREAMHOST most of domains on my dedi are up but everything under the user sercgroup on ds5648 are down.. the same was experienced with he last outage.
if you could look into the domains associated with sercgroup on ds5648 when i get chance would be great.. thanks.. hope the work is coming along well.. cheers..

@dawny — Thanks for looking into nslj.org. I’m switching my name to Alex2 because I think there are a few other people with the name Alex posting here. My other sites on bcrich seem to be working fine, including my sandbox at dev.nslj.org, so I’m not sure why nslj.org is still down.

Mine’s still not up. Says cannot connect to database.

I have the same problem as Bob. “Error establishing a database connection”

Site is still down!

For example, this is a website that is not working

You can access to http://www.bbchiardiluna.it/

but you CAN’T access here http://www.bbchiardiluna.it/tariffe.php where there is a connection to the database

March 27th, 2013 at 1:11 pmNot Impressed Says:

Wow, it sure is confusing to keep track of what we are & aren’t allowed to complain about. Thank goodness Bart is here to clear that up for all of us.

None of my domains I’ve checked so far are up, and I’ve received mysql connect errors from a cron job.

oh don;t get my wrong.. the other day when i lost a whole business day, my seo was effected and my office was at a standstill….. at that moment in time I was not happy at all… but after the moment passed … when i reflectively look at 10 years of service with them… they deserve another chance….

Glad that I can help. :)

ALL MY VPS’s ARE STILL DOWN

our site is still down as well, can you check on this please?

@Jim: Looks like your sites were just moved to a VPS server so it might be a combination of DNS propagation and the continuing maintenance period. I’ll have it checked on.
@Webb: While your ds5648 dedi is in the IRV datacenter, the MySQL server those sites are accessing are in the datacenter we’re doing maintenance on. I will get it looked into.
@Bob: Your MySQL server is in the affected datacenter as well. Having it looked in to.

I’m not able to reply directly to everybody above, but I am passing the information on to our team to be investigated. We’re aware that there are still some issues at this time and are working on addressing them.

site down… hey guys.. i have a few domains and most are working.. but promos.inscriptu.com is still down.. not sure if this is due to emergency maintenance or not since other sites are working

@ dawny … I setup the mysql on ds5648 in moved it from vps to the dedi ds5648 on my control panel all mysql shows its on ds5648 and also the user sercroup shows its on ds5648
the idea for that recent move was to have the sites on sercgroup and mysql on the same server… your now telling me it isnt ..??

@Webb: I just checked in your system config file for sercgroup.com quickly and saw that it was pointed to the mysql.broadcastm.com mysql hostname, which is on a VPS server in the datacenter undergoing maintenance.

ftp on raglanweather.co.nz is not working
raglan23.co.nz is down

I was having connection issued between the websites that connect to my mysql… i found out that the mysql and sites were in different datacentres… i thought.. much less chance of connection issues if I put the mysql in same datacentre as my sites.. and better still on the same box.. my panel shows in multiple places that my mysql and sites are on ds5648 in IRV… now im finding out my mysql is in LA andmy websites and mysql are still in different data centres

@danwy the website is http://www.bisousciao.com

Please help!

@danwy It for sure has been down for three hours, and though the site is up and down intermittently FTP & SSH have been consistently dead.

& dawny… ahh my bad.. sorry i thought that was switched over.. but i hasndt done that yet… the mysql i was concerned with is mysql.sercgroup.com and yes that seems to be up.. and that is the one that is on ds5648… sorry it was my mistake..

My site is still down as of 3:33PM Central time.

my site is still down as of 3:34PM Central time.

Luckily, it isn’t for making money.

sorry for the double post.

triple now – my bad.

@dawny.. yes sorry my mistake..

its just the mysql.broadcastm.com thats down which is on different account of mine.. and not on the server..

@Rodger Gallagher: both sites are up now and I restarted FTP on the server, which fixed it.

@Everybody else, please note the update to the status post. We are aware that there are some current networking issues with servers in the datacenter and there will be likely be intermittent issues until the maintenance window is over and we’ve resolved this post. We are working on those issues and I am passing on your examples on to be investigated.

Thanks danwy

Hello

we can’t connect to our phpMyAdmin at mysql.nubilahost.com

all sites connected to this host are still not working

My restaurant point of sale customers cannot get their licenses verified and cannot launch their software until they can access viewtouch.com

@Richard: I’ve tested your site again and it is still coming up. I tested your FTP now and it is working as well. If you’d like to submit a support request we can investigate this issue further for you.

Site has been down pretty much all day, no FTP either, any updates will be appreciated. Thanks.

@Rick, please see the updates on the status post at the top for the most recent information regarding this issue. I’ve passed on your site issues to be investigated, but I tested your FTP and it is working fine (you may need to use your servername instead of domain name though)

Our site is still down.

Site still down. (bisousciao.com)

Hey Dreamhost,

Any idea when mysql.supers-systems.net will be accessible again?

Dan

67.205.50.101 is still down, for over 4 hours now, not happy at all

@danwy Can you please let us know what IP ranges are down? Just to make sure our website (down) is not overlooked..
Thanks!

DH thank you for all your hard work, I eagerly wait for our site to return to live! Thanks again…

@bart
who cares if the service cost even $0.10? its the price DH has agreed to charge for its service. DH, in its marketing, claim they’re very reliable and this has not been the case in the last 6 months.
As a business person, I know to run a “lean” operation and when I look for a hosting service, I consider price and reliability. DH defeated it’s competitor’s because they promised they could provide reliability for the price. That’s why most of us choose them and that’s why we’re upset. I will not apologize for paying the little for the service!
If it is impossible to be reliable for the price, then DH should increase its monthly rates and see how it fares against its competitors!
Now if you’re not interested in my complaints, stay off the forum! the same applies to all other authors of post similar to bart’s.
PS: if my service wasn’t down, I wouldn’t even have time for this…….

Pinging [67.205.7.244] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

It’s kind of comical actually. Our website went down, but could be one of three things. Is it the fact that I switched DNS servers to CloudFlare? Is it Dreamhost’s maintenance? Or is it the giant DDoS attack sweeping the nation? Find out at eleven!

Some of my websites are up now .. but certain ones with Dedicated IPs are not.

Hello? Anybody from DH online?

@Bakery “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

@Daniel Horan: We don’t have an ETA yet, but keep an eye on our updates at the top for new information as we have it.
@Bakery: Unfortunately, I don’t have that information available, I’ve reported your site though and it won’t be overlooked.
@COPC: Forwarding that info on to be investigated.
@Alex: Passed your information on for further investigation.

@COPC sure but my ticket only gets generic/automated answers that say “everiis back to normal” so I am spsed to see that my website is still down, 3h later…in addition to yesterday.

Any way to find out what ranges are affected?

Ping your domain and you can see if it is working

is this the reason why i can’t upload media files? I keep getting HTTP Error. Been troubleshooting all morning but no source of the problem… I checked my other site and tried uploading to that site too, same problem! HTTP Error. Is there something wrong with the databases or something? Is that causing my problem? I wish I would’ve gotten at least some kind of email or something like “hey, guess what, our servers aren’t working correctly, you might be experiencing trouble”. for all i know this could be unrelated, but who’s to know?

@Paul…and if YOU aren’t happy reading OUR responses, then perhaps you should take your own advice and stay off these posts.

It’s not about the complaining…it’s about the people that have the ‘the sky is falling’ attitude, and ‘DH sucks’ yada yada. These people need to grow up just the slightest bit.

A problem occurred. Perhaps you’d rather that they don’t fix it?

@JayB: It doesn’t look like your issues are related to this outage. Both your web and MySQL servers are in our East Coast datacenter. If you submit a support request we will look into it as soon as possible.

okay, i checked my other site, it’s happening there as well, so that makes all 3 of my sites.

Ours came up, just to tease. Then went back down :o)

viewtouch.com 69.163.148.150 OK now.

Per our latest update, we’re making progress on this situation and are seeing more and more sites with service restored. We still have some work to do, but are seeing steady progress at this point.

Having trouble with my site, nerdwatch.com. Some pages are loading while others are not, or only partially loading. Also getting timeouts on loading some pages. I’m running WordPress latest version as my site content manager.

Ours is like a yo-yo right now, up then down (Rinse and Repeat)

raglan23.co.nz not working for ftp, site or wordpress login

It looks like connectivity between my webservers and my mysql server is very slow. All are in the LA datacenter: ps9411, ps159558 and psmysql9224. All my DB-driven sites (drupal based) are really slow while static or php-only sites are as fast as usual.
I ran some tests and queries that would normally be under 10ms are over 1000ms!

Still down. Customers calling, losing money. Please do something.

I still have a problem, but not sure if it is related or not. I can ssh into a domain that was down, but I cannot scp files out of that domain. scp isn’t accepting the same password that works for ssh. No idea why.

Opened ticket 5678795 on this about 2.5 hours ago and have heard nothing back yet. I know you folks are busy, and I am only trying to back up my site after a few minor changes, but it does seem odd that scp, which used to work, doesn’t now. Hopefully someone will let me know via the ticket what is going on, and whether or not it is related to the problems you’ve been facing.

All the detail is in the ticket.

@Francesco: We’re checking into your issue now.
@Rodger Gallagher: It looks like your processes are being killed by our process watcher for using too much RAM. Please contact support and we’ll be happy to look into it further for you.
@COPC: I’ve been testing your site for the last 20 minutes or so and it’s been loading consistently for me now.
@Ross: I tested your site and it is loading without issues for me now.
@jrpstonecarver: I’ll try to get someone to check into your ticket ASAP.

Still down. Rebooted VPS multiple times. Submitted tickets with no response. Site can’t pull assets from our VPS. Example at – http://cdaartauction.com/2012/lot/17617

Both websites are running, but slowly.

Still no email for our company. Website seems fine.

Still down, no website. Can’t FTP into static IP address or domain. Losing customer business!

@Michael: I checked your sites and they seem to be resolving correctly. The URL you’re posting loads, but is giving an error due to an unrelated problem with a VPS apache. I’m having that looked into.
@Etien: Your server is not one included in this maintenance. Please contact support for further assistance.
@Amanda: Can you give us details for your account? I don’t see the collectiveye.org domain in our system.

@Dan Edwards: Your VPS is having issues starting up. It seems unrelated to the outage, but we’re looking into it.

Thanks danwy – appreciate help getting this resolved.

@Michael: Your VPS needed a restart to fix an error with your rails site. Looks good now.
@Dan Edwards: Your site is now working again as well.

@danwy – any reason the 3 restarts I did of the VPS today via Panel, as recent as ~1.5 hours ago, didn’t do the trick?

@Michael: I’m not sure about that, sorry. I do see the reboot attempts, but not all of them completed. We were still in the middle of the networking issues when you attempted them, so it’s hard to say what was related to that and what wasn’t.

@danwy – no worries, thanks for the updates & help resolving this issue.

The SSH on my VPS was up, but I had to restart it before my http server would come up.

My webservers are not under maintenance and I’ve been experiencing many downtimes recently, the las 2 times support told me that “apache was restarted”.

This should get fixed asap.

@Umberto – I’m very sorry to hear that. Your web server is located in the Virginia Datacenter, and not affected by this outage. I see you have an open ticket, and we’ll get back to you as soon as we can. I appreciate your patience.

My servers PS10126 / psmysql10026 have not been able to connect to each other for nearly 3 hours now.

@Barry Wood – This has been fixed by our administrators. I am able to ping your VPS from your MySQL VPS now. Your sites are up and running well now. Thanks for your patience.

My problem continues, a WordPress-fed page will partially load, then pause for about 1-2 minutes, then finish loading. It’s like there’s a caching or buffering problem somewhere, the partial load always stops at the same place, pauses then continues. The connection is never broken, because it finishes eventually.

An example page: http://www.nerdwatch.com/things-ive-done/nerdtour-2003-usa/
loads up to half the content in the center column, then finishes and fills in the right-hand column. This behavior is constant regardless of browser.

@CptNerd – Although your web server is located in our Los Angeles Datacenter, this issue doesn’t appear to be related to the outage. Please submit a support ticket so that we can assist you further.

@Seohee, I’ll submit a ticket, but this behavior only started this afternoon, after you all began working on the LA datacenter issues.

Thanks @Seohee, it all looks good now.

Having connection issues on website

Not sure if this is propagation of service related. (New acct)

Appreciate any update

Thank you

@Grant – Our administrators are aware of the issue with your server and looking into it now. I appreciate your continued patience.

Still not seeing my website at all. DNS lookup shows no records. Weird.

Since this afternoon’s outage emails sent via phpmailer are not sent nor received anymore.
Any explanation? Is it due to the current maintenance undertaken?
Please let me know
thanks!

@Larry – It looks like the DNS for your site is showing correctly. It may take a few minutes to an hour or so for the updated DNS to update globally.

@Bakery – SMTP (outgoing mail) was affected by the maintenance. It should start working normally now — if not, please submit a support request!

everything is down on my end. email, websites, ftp, etc are all down…

@john – your issue is not maintenance related. If you post a domain on your account or ticket number from support we can check on that for you.

Opened trouble ticket, 18:04 PM PDT. I want to add info to the ticket but apparently you can’t do that. I’m assuming the “Contact Support” is how I enter a ticket. I wanted to add that I’ve tried a third network, and tried a couple of mobile devices and another browser, and still have nearly the same problem. The mobile devices just give me a “server has stopped sending data” message, Firefox tells me that, after it pauses, when the data starts back it says that the server is using an “unsupported compression algorithm”. The fact that it only started doing this this afternoon, is present on many different devices and networks, leads me to believe that it can’t be an issue with my client-side configuration. The only other possibility is something on the Dreamhost side, especially since it started when you all started working on your configuration.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)
5:35 pm

Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (March 26th, 2013 at 5:35 pm PST) by andrea

We are currently having connection issues in our US-West Data Center (Los Angeles, CA). We don’t have details yet, but our teams are already working on resolution. We’ll update this post as soon as we have more information.

Update March 27th 11:37am PDT: The problems appeared to be resolved then resurfaced, so with our CEOs approval, we have decided to go ahead with the emergency maintenance that was scheduled for tonight. Please see the status post above for details. We apologize for all the inconvenience this is causing.

Update March 27th 11:22am PDT: The issue from last night has resurfaced, our ultimate solution is the emergency maintenance which  we had planned tonight to fix it. We’re currently evaluating the situation, and will be making decisions in regards to pushing up the maintenance. We’re working on getting your service restored now.

If you have any questions, please contact support, so we can assist you. Again, we apologize for the inconvenience this has caused.

Update March 26th 8pm PDT:We are confident that all services are back up, and we are resolving this status post. Please note, based on our findings during this incident, we are conducting an emergency maintenance tomorrow evening. Please see the status post above for details.

If you have any questions, please contact support, so we can assist you. Again, we apologize for the inconvenience this has caused.

Update March 26th 7:30pm PDT: Our networking in that data center has been fully restored and we’re working on bringing back all internal services. We are keeping this post unresolved until we are fully operational again. Thank you again for your patience.

Update March 26th 7:02pm PDT: I’m happy to report that our services are slowly but surely coming back up. At this point, we are suspecting our Cisco router to be the cause of the incident. As the final portion of our Network Improvement series, this is the last piece of our core network infrastructure that remains. We’ve run into some problems getting it out of service in normal maintenance windows, before last week’s outage we were poised to finally pull the trigger and remove it but ended up having to postpone it yet again. It may end up being our best bet to perform that maintenance as an emergency measure tonight, we’ll keep everyone posted.

Update March 26th 6:40pm PDT: Unfortunately, I can’t report any material change in our status, just that our technical teams have been on-site and working on bringing services back up and continue investigating the exact cause of this incident. We’ve ruled out some anomolous activity already. We appreciate your patience and apologize for the inconvenience this is causing.

Update March 26th 6:10pm PDT: Our technical teams have been on-site and working on bringing services back up while investigating the exact cause of this incident. We appreciate your continued patience.

Update March 26th 5:45pm PDT: We are still working on finding the root cause of the issue and getting the services all back up. Please note, this affectes web, email and mysql services for customers in that location and also panel and webmail. Thank you for your patience while we get this resolved.

This entry was posted on Tuesday, March 26th, 2013 at 5:35 pm and is filed under General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

389 Responses to “Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)”

Assuming the web panel was up…

…if the web panel would even load. Not an option, guys.

Yes, well… the panel is down too, so I can’t see where my services are.

Here we go again…

WTF is going on this is getting ridiculous. I mean i have sites down, the dreamhost.com site is down and so is the control panel. WTF

In case anyone needs entertainment in the meantime:

https://twitter.com/search/realtime?q=dreamhost

I can’t get to the Dreamhost panel either. I can’t check or send email. I can browse any other site except for the sites that I am hosting. I hope this goes away really quickly!

I cannot even load the panel

When did the outage begin?

It’s pretty difficult to look something up on the Web Panel when the Web Panel won’t load! But I think it’s safe to assume that I’m affected by the service disruption since my website and email accounts are all down…

Yup, web panel does not load for me either. Site is down with the following message:

The connection has timed out

The server at http://www.mysite.com is taking too long to respond.

The site could be temporarily unavailable or too busy. Try again in a few
moments.
If you are unable to load any pages, check your computer’s network
connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

Seriously, nothing but problems with you people.

March 26th, 2013 at 5:39 pmTiago Nicastro Says:

Access down from Brazil. That’s include de panel.

Why is dreamhost down again? Many of my clients are dependent on these site.

toss in the towel

Email and the panel are out…Hoping you resolve this asap…This is an awful time to lose service.

Refund, for the last 10 years.

I thought my sites were down because I forgot to pay my bill. I guess this kind of make me feel better, except that I can’t make money to pay bills if my site is down. Quite a conundrum.

BOOOOO! At least now I know my sites are hosted in L.A. and not Irvine. Either way, this sucks!

My site went down at 7:20pm Central time. If you don’t already use it, sign up for a free account at pingdom.com, they e-mail you went your servers go down (and come back up.)

You pay less than 10 bucks a month for hosting, and you complain about 99% up time. You cheap babies. You want 100% up-time? Get a dedicated server you cheap fuck!

Not a good day to be having this problem guys. I can’t access my web panel either.

Trying to launch a major site and I almost went live an hour ago. What a mess that would have been. this kind of stuff is not encouraging :/

“You can check if you’re affected by this service disruption by looking up where your services are hosted in the DreamHost Web Panel.”

No, no we cannot.

Control Panel is down and all my sites are Timed Out.

OMG Second week in a ROW on my busiest/biggest night of the week? This is insane. I have really liked DH, but my members not being able to get to the site during the most important time of the week, 2 weeks in a row? This will not go over well at all

I feel sorry for anyone unfortunate enough to be using Dreamhost. I was just about to publish a breaking news article, instead I’m sitting here, watching other sites on Twitter getting the glory.

I feel genuinely ashamed to be using DH.

More like BadDreamhost.

Well I have a dedicated server and I’m down too!

Everything website is down (FTP, Email, etc etc etc), can’t get into Web Panel.

:(

Thank god I haven’t launched my site yet! Starting to have second thought about hosting with dream host…

Yep, just confirmed all my sites went down too.

This has actually been up/down several times this afternoon.

Here comes the “we’re under DDOS attack” lame excuse…

Panel is down, too. All sites timed out. Email inaccessible. Unhappy clients. Please resolve this quickly.

Does anyone know how difficult it is to migrate all of one’s Dreamhost content to a new host (whenever Dreamhost comes back up, that is)? Files are easy enough, but what about transferring all the database files and accounts to a better host Can PHPMyAdmin handle most of that successfully?

March 26th, 2013 at 5:47 pmLinux Torvalds Says:

“If you are experiencing a problem that has not been reported here, check our web panel for more information.”

Wait a minute…

S. Pat: It began at 8:16pm EST according to Pingdom’s log of my site’s uptime. They posted this article informing us of the problem at 8:35 pm, less than 20 minutes later. Personally, I’m in favor of giving them a bit of breathing room to figure out what’s causing the problem and deal with it rather than jumping on them for every brief outage.

I want to comment that I’m not here to gripe. I’ve had very good service and uptime from dreamhost for 7 years. Outages suck, but they are working on it. I didn’t expect to lose web panel access, though.

Dreamhost: the Laurel & Hardy of hosts.

“Cheap Dreamhost Webhosting Is Your Dealzmodo-Exclusive Deal of the Day”

Oh boy let me sign up! Oh wait, I can’t because their homepage is down.

It’s my birthday guys and gals. Not a good present. Please resolve ASAP, my clients are screaming. You are making us look bad.

March 26th, 2013 at 5:48 pmNot Jarandhei Says:

Jarandhei: personally, I am not.

I’ve been with dreamhost for 9 years and I gotta say they’ve been very good about uptime. I guess as they’ve grown they’ve had more issues and more people who really don’t understand the way webservers operate.

OMG WHY HAVE YOU RUINED JEFF’S BIRTHDAY!!?!?!

You cads, you cruel beasts!

Looks like constantly one or the other way, things are always down since few weeks

Dreamhost best service ever from 2001 to 2011. 2012 pretty bad. 2013 F-it!

Not again!! Always when I have AdWords campaigns running.

Why is Dreamhoststatus.com always up?

Dear Jeff, we will sing you happy birthday.

HAAAAAAAAAAAPPY BIRTHDAY TO YOU….

Maybe you all shouldn’t have offered new customers such a great deal on Gizmodo.com today…. too bad your current, and long standing loyal customers are paying full price to have non-functioning sites while you pile on new ones (and getting Gizmodo’ed)! Thanks!

Such an unprofessional way to run a business. Not even a panel?

Both panel and webmail down… Neither even pings.

HAPPY BIRTHDAY TOOOOO YOOOOOOO

I really hope it comes back quickly I have a slew of emails in my outbox and links to my website being advertised right now that don’t work.

Dreamhost, please provide more frequent status updates. Your lack of updates/info sucks.

I’m down. What is this, every other day for the last few weeks?

If this doesn’t improve very soon I’m switching hosts. There are way too many options for me to put up with this.

March 26th, 2013 at 5:51 pmone of DH user Says:

not again!
It’s been less than two weeks since last disaster.

Wow fail! My dreamhost panel went down when I was with a customer.

YOU LOOK LIKE A MONKEY

@Dennis — We keep DreamHostStatus.com at an external location so if there is a problem we still have a way to reach you!

My site is down as well as all email access. Very bad timing as I am at an important business moment and my inclusion in a major project is essential. This is very bad for my business and not the first time with DH.

Personally damaging and not encouraging!

I have had good support and uptime with DreamHost. Sure, this sucks, but oh well, I’ll have to attend to real life now. Darn.

AND DREAMHOST DOES TOO

@ DEMO: I kinda agree with you even though I’m wondering if that’s actually related to the outage as even dreamhost.com is down, are they on a shared service?

So this means we get refunds and/ or compensation for your lack of uptime right?

Just signed up for VPN because I thought this was over. Just traveled to work remotely at night with a contractor on my site and this? Really?

I’ve stuck with you for so long but this is really getting old.

Think of the children!

Think of JEFF!!

There was a time when I’d recommend you guys in a heartbeat. Awesome support, awesome features, and a great price.

I never understood why so many people claimed to have such bad experiences with you, but now I’m beginning to understand…

So disappointed…

Dreamhost.com is redirecting to “http://besthidden.info/go/tracker.php?t=54″

Things are getting weird.

Viacom site hosted on Dreamhost is down. You’ve made an extremely powerful enemy.

Really really bad timing. Money is bleeding out right now!

@Dave — We’re updating as fast as we have information! Please keep checking in!

@Dave: We will be providing updates to our status post every 20 minutes with any new information we have. Please stay tuned.

Guys, please. I been here for 7 years and this rarely happens.

For what they offer, DreamHost is a very good deal.

My sites are down too, but they are working on it.

Yesterday….
Dreamhost was a fun place to play…
But then they ruined Jeff’s birthday…
Oh I believe…
In yesterday…

@Victor — When we’re back, please send in a request for an uptime credit. Thanks!

So, on the 20th they had a power disruption which was fixed, and at the very same site they have an outage tonight. Want to bet they’re related? I wonder if someone plugged in just one too many coffeepots…

Kim Jong Un: 2, DreamHost: 0.

Just turned on CSPANN. Obama is addressing congress about the DH outage.

My God! it’s down again, I might lost a lot of users……
and even worse, we don’t know when it’s back……

@Zack – “Lighten up, Francis.”

@CptNerd – “Suck a cock”

2013 looks like a bad year for dreamhost (and us)..

All I can think of is BYE BYE SEO!

It’s a good thing Dreamhoststatus.com is not hosted on Dreamhost, or we’d all be screwed.

http://www.whoishostingthis.com/dreamhoststatus.com

Unbelievable! My site is down again. Seriously guys fix the LA Data center issues once and for all.

March 26th, 2013 at 6:00 pmPGP_Protector Says:

Also taking down SVN Services !!!
So No updating my Code

Another time all my sites down. It’s not fair. I don’t know what I’m going to do. It’s a catastrophic situation for me. I will lose a lot of clients and users. It’s dramatic.

once again dreamhost letting us down. this is the nightmarehost ladies and gentlemen.

Come on guys, even shorter one whois.sc/dreamhoststatus.com seriously, get with your web knowledge

I’m about 7.5 minutes away from running out of porn.

THEN WHAT?!?!

“It’s dramatic.”

Uh oh, State Premier is about to launch a new project this morning (earlier than we thought he would) and we scrambled to pull together the content and the site I was working on for the client has gone down.

still down. what’s going on.

Sorry to hear that Katie Hannan.

I just don’t get it…what economy of scale do you need to be before you invest properly

in backup systems????

The message is : greed comes first, customers come distant second

Doesn’t sound like a sustainable model.

March 26th, 2013 at 6:05 pmBangkok Moon Says:

UN is convening an emergency session of the Security Council.

Subject: Dreamhost outage.

@ Matt: *this* is shorter
http://waa.ai/SBv

Nice! Thanks Marco!

really annoyed right now… just had a major update and sent a fair bit of invites a few hours ago… all my websites are gone…

http://who.is/whois-ip/ip-address/96.126.109.82

Doesn’t that mean dreamhoststatus.com is hosted on something called Linode?

The worst part of Dreamhost drops, the retarded end user.

“What do you mean our content is hosted outside the building?!”

An update every 15 minutes would help for a major issue like this. Lack of info sucks.

not sure how they are working on a resolution when they don’t know what the problem seems to be

East coast of Australia down…in the middle of the day ?!?!?

Perhaps if you guys spent less time crafting cutesy newsletters that read like it was written by junior-high school kids and more time working on your hosting…

lets go to amazon

:D

No way again, come on!!

@ Doug: no, it doesn’t.

This must all be because I upgraded from shared to VPS account. I’ve obviously overloaded the system! Sorry everyone, it’s all my fault. I’ve been with Dreamhost for almost 10 years, perhaps it’s time to move :(

Why have comments?

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

This is totally stuffed, we are in Australia and its mid day with this crap happening we can’t access our emails for our business. Totally left us in the dark. Will not renew with dream host again.

March 26th, 2013 at 6:11 pmBangkok Moon Says:

Cliff, you power-sucking abuser! It’s all Cliff’s fault for driving us off a cliff!

Really again?

Because they’re amusing.

‘The Three Stooges are currently trying to resolve this issue for Dreamhost’

Downtime is a pain, but to have the webpanel completely MIA? When you can’t even run your own services, I question why I just re-upped for another 3 years.

March 26th, 2013 at 6:13 pmFresh Ice Tra Says:

Got a tweet saying DH would comp me with a free year of hosting.

please fix – we are trying to broadcast live

Hey, if you’re in Australia, maybe you should get Australian hosting. Just sayin’.

@Dave: During large outages such as this, we do actively monitor the comments. Please note that Elizabeth and I are watching them as we speak. If you require assistance, we’ll do what we can to help.

Well isn’t my face red then.

You are right, just been with dreamhost for over 8 years now. But its time to move on

WK -> I’m in Australia and Australian hosting is terrible. It’s extremely expensive and you get terrible data limits. Hosting overseas is the only viable option.

Oh man what a coincidence. I literally switched service providers for my personal site to Dreamhost TODAY, and this happens. I’ve been using them for 2-3 years for our business without issue. This however…sucks royally.

Thank god for CloudFlare. After the last outage I turned it on and I am happy I did. I’m using my website to also host files for some of my servers, if it weren’t for CloudFlare my whole company would be down right now!

I highly recommend using CloudFlare. It has slowed somethings down a bit, but in times like this it really really comes in handy!

No joda!!!! hasta cuando con esto ? no ha pasado ni el mes y ya se cayo de nuevo ?

esto es una seña que nos dice a todos: Salgan corriendo para otro proveedor!

Twice in one week. Been with this company for maybe 7-8 years, but now am seriously considering moving on.

What other international hosting providers do you guys recommend for businesses

Again…

Eh, life goes on, they’ll be back up. I checked my outage list and my update for the past year is 99.93%, so I’m not complaining. Stuff happens.

I turned on CloudFare and all my sites are still down. What a nightmare…and what’s worse…my dedicated server is Irvine…go figure.

Uptime lol

Nice to see all the upgrades over the past year have removed the single-points-of-failure (lol).

I’m seriously considering BlueHost now. I was just about to signup some new customers but I have totally lost my confidence in Dreamhost and I’ll be sending the customers elsewhere.

Will the emails not being received right now actually be received?

Also yes, Dreamhost is getting stagnant now….

So… where is dreamhoststatus.com hosted? Perhaps I should switch to them…

I’m losing money right now, goddamnit!

@Ralph at Dream host – I can’t believe a week later and here we are again. Ever since I moved from shared to VPS my site has been down down down. I would appreciate a status update. Hoping for a quick fix.
I can’t have my customers loose faith in my site.

Any word on when it will be back up? My site is completely down.

@Sara — We’re working on it as we speak!!!

I’m from Brazil, I can’t access nothing, neither panel. I hope this issue will be resolved fast.

March 26th, 2013 at 6:21 pmFresh Ice Tra Says:

@Seohee – but what about Jeff’s birthday.

After years and years of great service this is a first.
hope you get it fixed soon.

This is typical, been down 5 times for over 12 hours in the last 12 months. Def time to move away from them. Worst hosting we have ever used.

@Alec — Emails sent to a DreamHost email account right now will either bounce back as undeliverable to their senders, or the sending server will save it to try again, depending on how the sending server is set up.

Really people? Crap happens. Get over yourselves. I have had Dreamhost for years and have had accounts elsewhere as well.Guess what. I have been moving them all over to Dreamhost because of better service. You want to whine like a little girl, go to GoDaddy where the rest of the people who wet the bed are. Less than an hour and people are wanting to light torches and storm the damn castle.

It’s easy to place blame. Give them a break. You wouldn’t get it until you’ve run datacenters. Been there. It’s not as simple as flipping a switch

March 26th, 2013 at 6:22 pmBenedict XVI Says:

This wouldn’t have happened if I were still Pope.

right when the site comes back up I’m transferring to a new host. Anyone know of an actually good, reliable host?

I noticed that SVN was still up and running…. weird right.
I was kind of hoping maybe there’s some sort or cache or buffer in operation…..

@Sarah — We don’t have an ETA yet other than “As soon as possible!”

WE ALL GONNA DIIIIIIIIE!

Sara -> Yes me too. It’s strange, the uptime seemed pretty good on Shared for me. I moved to VPS and down it goes. It’s obviously just bad timing, but paying more for a service and then having down time like this really hurts.

Even dreamhost is down, I think all our sites are gone

March 26th, 2013 at 6:23 pmDH Weekly Major Outage Says:

Welcome to the weekly edition of:

Connection issues in US-West Data Center (Los Angeles, CA)…this week, we will be featuring the following:

5. How to lose your clients enmasse by using DreamHost hosting services
4. How to publish 99.9% uptime and yet still be down what seems like 30% of the time
3. How to figure out a way to engineer a ‘reverse redundancy’ data model
2. How to continuously recycle customers & yet customers continue to come
1. Yes, #1 folks, is the age old question, if a webhost’s own site goes down and nobody has the means to report it, is the webhost really down at all (see the 99.9% uptime) ;-P

Soooo…how do we get credit for downtime guarantee if we can’t even put in a ticket?

NOT AGAIN! OMG! FIX IT FAST!

I can’t believe… AGAIN ?! My customers are pissing on me.

Wait…I got passed over when they were handing out torches.

Where do I get one?

I’m on the east coast server, but it still has killed webmail and the control panel. Sites seem to be loading. Is the reliability getting worse? Seems that way now that I am actually relying on it.

second major outage this this month!

complete downtime once again!

we might as well laugh because this is utterly pathetic…

Hey DH, maybe start at the power socket and work back?

There is a surprise another OUTAGE! Dreamhost sucks time to move.

Wakey-wakey…

“SECURE !!!” systems are being hacked around the world….the amount of downtime here is becoming bothersome. I have twelve e-mail accounts going through here, three money making sites. What gives, guys? Do we need a shake-up in management?

Mitchell – > You’ll have this with any host. If you find one that has a history of 100% uptime, let me know.

@Kate — Once this is resolved, please send in a ticket with your request. Thanks!

They got lifehacker’d. Lol
http://lifehacker.com/5992460/cheap-dreamhost-webhosting-is-your-dealzmodo+exclusive-deal-of-the-day?tag=deals

Anyways, was just going to renew my domains when I couldn’t load the site. They were just giving a deal to NEW CUSTOMERS on web hosting. You can bet a lot of them will cancel due to this outage. Also, current customers should be able to take advantage of the $20 for a year deal especially with this outage. Ridiculous. What happened to the 99% uptime? Just came to rant and give my 2 cents like everyone else.

@Benedict XVI lol

Can I get in on some of that golden shower action, Andre?

send him back home –> When you get your CCNA and can offer to help get them back, feel free to speak up. Must be nice to be an anonymous jerk on the net..

@Kevin – No datacenters aren’t easy to manage, but if done properly then they wouldn’t be down… Why do you think google is never down, it is properly managed – unlike dreamhost

March 26th, 2013 at 6:27 pmSEO going to hell Says:

DreamGhost?

I am currently on a new customer plan right now and I am seriously doubting if I will renew with DreamHost

March 26th, 2013 at 6:29 pmsend him back home Says:

haha just checking to see if they were editing these comments. yes they are.

Who hosts dreamhoststatus.com Maybe we should switch to them. ;-)

Come on people!!!! I have super-duper important stuff on my Dreamhost sites and the fact that I can’t access them means major disaster for me and millions of my customers!!! I have been running my multi-million dollar business off of my $10/month Dreamhost account and now it is all swirling down the toilet!!! I came up with a brilliant plan to end world hunger but I am storing it on a Dreamhost server and now people will starve!!! NASA depends on my analysis to keep the International Space Station in orbit, but all my data are on Dreamhost!!! Hospitals around the world are waiting for me to send them the cure for cancer that I came up with, but I can’t access my email!!! You must do something about this right now!!! I demand it!!!!

Here we go again!!!

Somebody tell the cleaning lady to put the plug back in when she’s done vacuuming!

NightmareHost

@send him back — Only yours. I’m here to help — useless vulgarity isn’t necessary.

Mitchell – Google has outages all the time, but they have multiple data centers, and fail over between them. They have six in the US alone. Not to mention their capacity is 100 fold more than any single web hosting company. I ran 8 data centers (well) and you can’t predict or prevent every issue. If you think it’s an issue of being done properly – try applying for a job, then we’ll contact you when there is an issue.

Well, I have about 50 sites hosted with DH and I’m not crying my heart out because of a failure that can happen to anyone. I mean, get a hold to yourselves! it’s true they got a serious trouble there but then again, at least as soon as they noticed they started working on that. I don’t see how you blame them for doing their best in getting everything back.

March 26th, 2013 at 6:31 pmelliot smith Says:

it’s not the downtime that’s frustrating. it’s the complete lack of transparency on the part of DreamHost. It’s the failure to publish uptime history. If dreamhost published uptime history, then users would be able to see that the downtime isn’t all that frequent. It’s like a cop, never there when you need it, there when you don’t. But dreamhost can’t figure that out. dreamhost can’t innovate. it’s sad.

Is email sent to dreamhost servers at least being queued? If not, why not?

Jon says:

Nice to see all the upgrades over the past year have removed the single-points-of-failure (lol).

Thanks guys for working hard to get the servers back up. I’ve had my site for 15 years at various hosts. In my experience, dreamhost is one of the better hosts for the price. But I am curious about this downtime ticket price thing? What’s that about? $20 a year? I’d like to partake!

March 26th, 2013 at 6:33 pmRodrigo Silva Says:

I’ll be back.

March 26th, 2013 at 6:33 pmsend him back home Says:

bethiepoo: useless vulgarity would demonstrate that you are not editing these comments to the benefit of dreamhost. since you use this site as a sales tool (“look how open we are about our problems”) editing them becomes an important phenomenon. see you on yelp.

March 26th, 2013 at 6:34 pmelliot smith Says:

this is how all dh customers feel right about now:
http://youtu.be/N3JFwd1bk4Q?t=1m23s

Just when I thought things were looking better…..

Everything is down — no websites, no e-mail, no panel. On the other hand, Dreamhost’s own website is also apparently down.

I wonder who hosts this site?

Does this Kevin person work at Dreamhost?! Way to manage your clients, dude.

As someone earlier stated, you guys pay $10 a month for hosting, and 99%+ uptime just isn’t good enough for you. WTF…

Quick! Somebody please make a “Hitler Reacts to the latest Dreamhost Outage” video and post it on YouTube!

Hey boo-ums, I have a dedicated server and I’m still down. There’s a real issue – don’t diminish it.

Been with DH for many years, not sure what is up with the unreliability lately. Needs to stop.

Eli – No, I don’t but I manage data centers for another company.

If anyone at all knows what they’re doing on this board – you may want to investigate. It looks like it’s a Cogent issue, not necessarily Dreamhost.

@Eli — We haven’t forgotten you! Update in a few seconds

Dwayne, this site is hosted somewhere else for exactly this reason — when everything goes belly-up, they can still tell you about it.

Ain’t the first outage, won’t be the last. After being with these guys about 10 years, you realize it just happens sometimes. :)

Very bad timing since my hosting renewal bill arrived today. You want $120 and newbies are getting the same service for $20? I think it really might be time to go after 7+ years of this.

March 26th, 2013 at 6:38 pmsend him back home Says:

un-edited thoughts about dreamhost from their customers.
lots of discount offers from competitors happening right now:

https://twitter.com/search/realtime?q=dreamhost

March 26th, 2013 at 6:39 pmSEO going to hell Says:

People saying that DH has 99+% uptime are naively repeating what DH says. Pathetic. Yeah, and McNuggets are made with the best grass-fed chickens.

“I have a DREAM: Some day it’ll be working!!!!!!

Kevin – Google’s outages usually have to do with new software implementations. I’m not saying any data center can keep 100% uptime, but it would be nice if we knew the problem. Every hosting company has an x% uptime, some have more, some have less. This is due to nothing other than the lack of redundancy and thought when keeping these issues in check

not all of of us pay $10 a month, some of us are hosting real businesses with real requirements and expectations, and real customers, for us and the price we pay 99% up-time is not an unreasonable request

@send him back home – They don’t have to allow comments on this page at all. The fact that they do, shows a certain amount of transparency don’t you think? I’m with @Elizabeth in that vulgarity has no place here. I have absolutely no doubt that they are working very hard to rectify the problem….and vulgarity helps how exactly?

March 26th, 2013 at 6:40 pmDH Weekly Major Outage Says:

So they created their very own DDOS attack by launching a ridiculous Dealzmodo campaign that will continue to send the professional web designers out the door to be replaced by bandwidth whoring P2Ps & N00Bs, great job, DH!!

I gotta tell you, since they brought this on themselves this time, I am truly pi55ed off!!!!

@Elizabeth my friends sites who had shared hosting seem to be up and running – is the issue with VPS and Shared?
I know their sites are hosted at the LA data center because they were down last week.
I’m paying more to get more down time…. ??!!

@Elizabeth I meant– is the issue with VPS and dedicated servers only?

@Sara — The issue is with a specific data center. Your friends’ sites may be at one of our two other data centers.

My site is up and working, but my mail isn’t. Mail affected?

March 26th, 2013 at 6:42 pmDH Weekly Major Outage Says:

@Daybo (Obvious DreamHost employee), please go back to fixing the FAQing problem, instead of trolling the comment page for PR reasons

I find it kinda funny that DreamHost’s site doesn’t have any kind of back up system

Typically, I post concerns over down time to the web panel but as that is not possible right now I figured I should post it here. To start off with, I should say that I have been using DH for about 7 years, and I feel the need to be patient when dealing with them over a low cost shared hosting plan.

But, I recently added a VPS account (about a month ago) for a new site I’m working on where reliability and responsiveness will matter more. I’ve been entering a lot of data as I continue to prepare for announcing this site on numerous boards, and as I can’t get any work on the site done tonight, I’m wondering if DH is a good solution for this new project.

Does anyone know if DH offers any sort of failover service for a reasonable price to prevent these kinds of things from crippling my site (I am using cloudflare, but that is not really enough)?

This is getting to be a joke. I’m in my 2nd year with Dreamhost and I’m really starting to have second thoughts. In my time, there has been several large outages like this one, and untold small outages: the day to day loads that DH can’t handle.

@Daybo – why else would they allow comments? So all the users can talk about how happy they. Come on now

March 26th, 2013 at 6:42 pmsend him back home Says:

over the past week, dreamhost has had 93% uptime. anyone defending this is a moron.
kevin is going to look like a complete tool when the idiot ceo comes in here to apologise
for being an idiot ceo again.

@Kevin – google for “CCNA dumps”. No one’s impressed with your baby cert.

While we wait let’s all say how long we’ve been with them.

February 2001.

@harlequinjunkie – It will be affecting some VPS and shared if they are located in the LA datacenter. Again, we apologize for the trouble.

This is a bummer. Had a meeting scheduled for 6:00 PM Started at 6:15 and no website. :(

Can you please provide another update????? More communication would be better!

“” Elizabeth Says:
@Sara — The issue is with a specific data center. Your friends’ sites may be at one of our two other data centers.””

How many data centers went down last wednesday?
They were hosted at the center that was down last week but are working fine right now

@Ken R @send him back home
You don’t have any clue what you are talking about since you don’t have your CCNA – complete sarcasm

I have a dream…that hosting companies will not be judged on the quality of their newsletter but on the percentage of their uptime…

March 26th, 2013 at 6:45 pmsend him back home Says:

daybo: shut up your cunt mouth

@Dave – An update is going up shortly. As mentioned, we try and provide you with updates every 20 minutes. Please note we are doing our best to provide you with any information we receive, but we also need to make sure that attention to fixing the issue itself is our priority. Thanks for your patience.

March 26th, 2013 at 6:46 pmSounds legit Says:

A “datacenter manager” from “another company” is here making excuses for DreamHost.

Sounds legit.

Mitchell – Most of the time it’s hardware failure. Do you have any idea how many individual machines they run in a cluster? What their thermal management looks like?

Yeah – I can take a look at the Google App Status board, or techcrunch and say it’s software. My son can make that assumption from searching the web.

PDU’s fail. Hard drives fail. Individual machines in clusters fail. It happens in EVERY data center, even Google. Hell, even a core switch can take down racks of servers without properly failing over.

Google’s data centers operate on a large enough scale that it can shift redundancy from hardware to software. There’s so much hardware that it can use software to identify failures and route around them, even at the cluster level. That’s the benefit of having a billion + dollar company.

Dreamhost – I feel ya guys. Please know that some of us still care and know you’re working hard.

@Sara — We have three data centers, US-West (Los Angeles), US-West (Irvine), and US-East.

Come on Dreamhost, get the email side of things to work atleast. Its 12:40pm here in Australia and we have no email access in the office.

Ah! Was wondering why my emails were in limbo. Hope you get it resolved soon.

I have been with dreamhost for over 10 years. They have incredible service. No hosting company is without its problems, and especially not FOR the price.

Don’t listen to the whiners who love drama and complaining (they are wetting their choppers over this.)

I vote dreamhoststatus.com for DHSOTM!

Sounds legit. – > Least I’m using my real name and am willing to go to bat for something I believe in rather than hiding behind a ‘street name’ ..

March 26th, 2013 at 6:47 pmDH Weekly Major Outage Says:

10 75 ms 64 ms 61 ms te0-1-0-2.ccr21.iah01.atlas.cogentco.com [154.54
.2.82]
11 88 ms 90 ms 89 ms te0-2-0-5.ccr21.lax01.atlas.cogentco.com [154.54
.0.233]
12 113 ms 332 ms 180 ms te7-1.mag01.lax01.atlas.cogentco.com [154.54.28.
142]
13 * * * Request timed out.

Please update, as our cloudflare cache did not function properly =( Unfortunately we thought it was a sufficient preventive tactic. As a startup, we can’t afford the downtime with dreamhost anymore.

Hopefully, they’ll compensate for it, but their trust has definitely taken a hit.

So all email is down?

It’s 12:40pm in Australia and this is effecting my business emails. Good luck with resolving the issue Dreamhost! hope its fixed up quickly.

I am paying dreamhost $1,500 per month for VPS and Dedicated .. and I have a new business in Australia… one day this month.. DH went down for full business day and then up for 4 hours then back down for rest of the night..

This is second business day in a month… considering 30 staff in my office cannot work… Lost wages and business is in the tens of thousands….

I guess what thy mean by 99% uptime is 1 day ur of every hundred they are down .. well its 2 days this month so far….. after 10+ years DH is making it hard to trust them anymore.. .
feels like one of these times when they have a big enough outage.. they just going to say bugger it.. too hard…

Rabble!

March 26th, 2013 at 6:48 pmsend him back home Says:

the idea that someone needs a CCNA in order to effectively
host their site on dreamhost is a real laugh.

Hang in there, DH staff! :-)

I don’t normally even bother to post but the entire Dream Host site being down..wow. I was talking with the Kickstarter folk and now..voila no site. Yeah this looks good. Customer credits would be nice. I have NEVER asked for one in 10 years. But now….let’s talk about it DH.

I’m paying DH $116,000 a month to host my cute cat pics.

send him back home –> No one said I needed it. Earned it when I was 17 and just maintained it for 25 years. What’s a real laugh is that you can point fingers and can’t use you’re real name or offer any possibilities other than BLAMING dreamhost.

$10 per month since 2006.
Website is up.
email is down.
I frequently consider switching, but I know it’s a lot of work to switch.

Good job…back up. Thanks guys.

Kevin – Yes, hard drives do fail and yes I do know what thermal management looks like I have been to many data centers. Any decent data center should have the redundancy so that if there is a physical problem it can be switched over. The only reason I would not blame dreamhost is if they were being attacked by hackers (I would probably get on to them for their lack of security in that case) Now while we wait for the servers to come back online I will go get my CCNA…

My sites have been slow all day. Was chatting w/ DH earlier today and they were trying to tell me it’s because I don’t have enough caching. Doesn’t make sense when they worked fine yesterday. Guess the real problem is the connectivity issues mentioned in this post since my sites are at Irvine. I was going to upgrade to your dedicated servers but now I’m not so sure you deserve my business. The fact that your chat rep tried to blame your issues on me really irks me and wasted hours of my time today trying to track down a non-existent issue on my end. Of course, I still haven’t heard back from your representative (over 3 hours now) since I pointed out that the lack of caching argument made no sense when the sites were working fine before today.

@Vince – Webmail is down at this time, including mail located at this datacenter. However, not all email service is down.

Most of you complaining about services being down and affecting your business, dont have a business. Stop lying.

@Elizabeth My site is up, hopefully it stays up.

@jen — This issue isn’t with Irvine, so if your sites are at Irvine, they aren’t affected by this.

SEO going to hell Says:
“People saying that DH has 99+% uptime are naively repeating what DH says. Pathetic. Yeah, and McNuggets are made with the best grass-fed chickens.”

They are made with grass-fed chickens. Note the ads don’t say which _part_ of the chickens they are made from.

I really don’t get why people complain so much. Sure, I can understand who’s running a business but even in that case don’t you guys have a life? Don’t you guys realize this is just part of our life, and not the most important part by the way?
Instead of complaining here why don’t you go and hug your mother, brother, wife, husband, whatever…?

And for all those guys who keep saying it’s time to move, maybe Bluehost, makes a favor yourself and google “bluehost down”.
I’m not saying Bluehost is bad, a client of mine is on bluehost and she is a happy client. They have their problem too as everyone else.

Just calm down, breathe and take advantage of the moment, go outside and walk.

March 26th, 2013 at 6:53 pmDH Weekly Major Outage Says:

Matt Says:

While we wait let’s all say how long we’ve been with them.

February 2001.

/me since January 2004, however, I moved my business sites to Arvixe after a really bad run they had back in 2009….kept hoping it would get better and that I would bring my high volume sites back, 4 years and counting :(

how can i access the dreamhost panel and change my dns.
my domain was registered at dreamhost.
please.

15 minutes till the media launch for the site i worked on. i’ve moved the site off dreamhost and now i can only wait for either DNS to update or dreamhost to come back online. unbelievably bad timing, sounds cliche but this seriously could not have come at a worse time.

Since 7 year the most frequent words at DH status are “We don’t have details” and “We apologize for any downtime”

Mitchell – They have three (3) datacenters. Not everyone, even a $10 a month hosting company can have 100% redundancy – hence my point on only places like Google being able to do it.

They’re still trying to pinpoint the problem. I’d like to see some encouragement to their staff, rather than the lines and lines of endless bitching…. Maybe you could support that cause?

We’ve been with Dreamhost for several years and this is the first major downage of this magnitude…so I’m more than willing to give them a wide berth here. This is obviously something pretty nasty, so patience definitely needs to be in play. Standing by, DH.

March 26th, 2013 at 6:55 pmSEO going to hell Says:

“We’ve been with Dreamhost for several years and this is the first major downage of this magnitude”

This is a lie.

Marco – Thank you. At least you get it ..

“connection issues” means that the people they buy internet with are having an outage. Not the dreamhost servers. Huge difference.

Of course, they could get more than one internet connection, or more than one hosting location with all sites mirroring each other. They can’t, however, fix the phone company. Complaining that their servers and control panel are down is like complaining about a store not having working cash registers during a power outage.

Seriously? Absolute bullshit. Get your shit together.

I’m always a bit saddened when these outtages happen and we get to see civilized behaviour completely fall by the wayside in an instant. If you don’t want any downtime from your hosting company, keep track of the Dreamhost outtages, if it gets beyond your tolerance threshold then simply move to another hosting company. Express yourself with you wallet but please try to behave like a relatively well balanced adult.

We are talking about technology here, NO technology is perfect all the time…. ever.

Happy to be a Dreamhost customer and happy to use my full name.
Take care everyone
James Sommer

@Sara – The issue is with some VPS, shared, MySQL, webmail, and email located at this datacenter, Dreamhost.com, and the panel as well. No dedicated machines are affected.

Call it a lie if you want. I’m just going by my experience. Believe me, I have no dog in the fight…just a customer like the rest of you.

March 26th, 2013 at 6:58 pmSEO going to hell Says:

@Great Big Radio right – then I will call it an “inexact fact”.

I have VPS and my site just went Back up. DH panel no luck yet – Check your sites

@scot – I’m afraid you won’t be able to access the panel to make your DNS changes until the issue is resolved and the panel is back up.

March 26th, 2013 at 6:59 pmsend him back home Says:

dude, they were down for almost 15 hours last week. you’re not paying attention.

or they could just be throwing out the phrase “connection issues” when the connection issue is on their side of the big internet plug in the wall.

My site is still down, of course they will be working as hard as possible on this because it involves everyone’s sites!

http://www.sermonindex.net

Hi guys – thanks for maintaining this website and keeping us in the loop. Good luck getting service restored quickly!

I JUST signed up a few days ago. Not sold yet. At all.

Very disappointed.

Don’t worry dreamhost, I’m not leaving. I like you guys. Keep up the good work.

@Seohee I moved from Shared to VPS at the beginning of March…it has been one problem after another ever since. Prior to that I didn’t have any major issues.

I joined Dreamhost in 1912. Service was better back then.

March 26th, 2013 at 7:00 pmDH Weekly Major Outage Says:

@Marco,

step away from the Kool-Aid stand…..

It is terribly patronizing of you to make those ridiculous statements to peeps who are paying very good money for reliable webhosting….some of the blokes posting these comments are acquaintances of mine and they are paying into the $1000s of US dollars a month…not cool to minimize their concerns….

You get what you pay for is too true, I actually stopped using DH to host my actual business sites years ago when I realized 99% uptime meant they go down about once a week or so. . .
I recommend to anyone with the knowledge, to maintain their own server. I just use DH as a DNS server, FTP server (for file transfers) and e-mail server. Their DNS server has never gone down as far as I’ve noticed, and this seems to be their longest e-mail hiccup in recent history that I recall.

Honestly I’ve been running several business websites (serious ones with terabytes of DB’s an a F*Ton of traffic) off of a mac mini in my basement for 4 years now and other than 1 day of downtime because verizon had a big boo boo I’m always online.

Downtime, etc, is all well and good for shared hosting on VMs – but VPS hosting? Dedicated hosting? Multiple issues in the same month/quarter? And I’m about to put a real client on this server, with real uptime needs. Or am I?

I’M WATCHING GAY PORN!!!

Kevin – These comments are completely pointless. Who are we going to encourage? All of the technicians that are reading this page? The only people from dreamhost reading this page are moderators who are not at all working to fix this problem. Let me give it a shot “Let’s go dream host” Make sure the ethernet is plugged in!

March 26th, 2013 at 7:01 pmAsia business Says:

This is a serious disruption to our business here in Asia. This is at the beginning of our work day.

Someone earlier wondered why this is such a big deal. Because we are in the middle of making a deal that will impact our company and our staff and their families.
And this outage is seriously impacting that.

That is why.

For those that think this is acceptable, you most likely are not responsible to run a business and can not imagine this kind of scenario.

Is this receiving all your attention Dreamhost, because it certainly has all of ours.

@Sara – My apologies for the downtime you’ve experienced. I am truly sorry to hear that. Please note that you can contact support (once the panel is accessible again) and request downtime credit for your trouble.

March 26th, 2013 at 7:02 pmJonny Rocket Says:

Best Web Hosting =)
http://lifehacker.com/5911651/five-best-web-hosting-companies

I love DreamHost, all you crybabies can cry some more…

just chiming in on my sites being down and not being able to connect to Dreamhost.com at all.

Well that’s disturbing… apparently someone just posted as me that they were watching gay porn. This was not posted by me!

March 26th, 2013 at 7:04 pm10 year user Says:

Going on 10 years w/ DH – have recommended to thousands of clients. I can’t run a business when my sites/email are down – once in awhile, yeah, it happens. But it’s been happening WAY too often the last several months.

DH – you need to directly address these issues with your current client base and offer credits, as the very least.

March 26th, 2013 at 7:04 pmDH Weekly Major Outage Says:

@Drew

Love getting back to the basics :)

FRIENDS! TODAY i move over to a new hosting service!!!!
—THREE TIMES i the last two months there have been major outages.
—WEEKLY my VPS hosting server stalls and has to be rebooted (when I discover sites are down)
—my sites have been HACKED easily a dozen times since I have been on the Dreamhost servers.. which I believe is due to their poor security.
—Very poor spam control (compared to earthlink.net for example
Just such a terrible subpar hosting company.
Bye bye and good riddance!

Thanks for the list of other more reliable hosters Johhny Rocket…

nice troll attempt backfires!

@10 year user – Absolutely. Please contact support once the panel is accessible again to request credit for the downtime.

Updates every 15 min. please.

unverified real name is a believer who bats.

Years ago I had a leased server at RackShack (note: not RackSpace), and one of their datacenters had their backup generator explode, blowing a hole in the wall of the building and naturally stopping a whole bunch of servers, many of them damaged in the blast. Try explaining to your customers that their website isn’t up for weeks because a datacenter blew up. This is nothing, and they’ll have the servers back up before long. The people who started talking about leaving when the outage was a few minutes old will never leave because they need the outlet.

how can I switch to the Irvine data center? I’m in San Diego so that makes sense.

again?

@James – I’m FRESH OUT! I *RELY* on Dreamhost for my critical porn needs.

This is unacceptable.

why do i pay extra for a Private Server exactly?

same here.
I hope this gets resolved in less than an hour…

they try and give you guys updates every 20 mins and you’re impatient enough to complain they aren’t updating enough? get a grip people.

March 26th, 2013 at 7:09 pmelliot smith Says:

In many years, I havent ever seen DH improve their communication vis-a-vis downtime….

“Sorry for the inconvenience, the issue will be resolved in F— Y– minutes, the uptime for your virtual server account is F— Y– %, in case you are curious, the cause of our last downtime issue was an issue. Thanks for watching, good night folks!”

^^^^DHs idea of transparent communication.

March 26th, 2013 at 7:09 pmJames Sommer Says:

@Faith – I keep the gay stuff to myself, but I can share some quality midget/bat guano shorts if you like.

Good luck with the work DreamHost !

I churned through probably a dozen hosts, from big and expensive national providers, to small mom and pop local shops, not to mention doing it myself. Then, in 1994, I signed on with DreamHost. I’ve never regretted it since. The prices have gone down (incredible!), the service has remained high (excellent!), and what I get from them keeps increasing (amazing!). So, I take it all with a grain of salt. Sure, there may be some down time, but it’s only a Website. As someone else said above, it’s not like the International Space Station or a cure for cancer is hinging on our sites being up every second of the day. If it were, you’d pay through the nose for a site that guarantees an additional 9 or two (99.999 or 99.9999% uptime). However, each 9 costs about 10x more than the last, if not even more. So don’t bitch and moan. Either pony up a lot more money for rock solid hosting (which still never reaches 100%), or cut the admins a break. I think they are awesome, and I appreciate all they do for us. They certainly have made my life easier.

Any Dreamhost moderator… is there a way to find the identity or IP address of the person(s) that made a post using my name? Just curious.

Thank you,
James Sommer

@Justin — When we’re live again, send us a support msg and ask to move to Irvine

my site just came back up! thanks for the hard work dreamhost :)

e-mail’s Back Yay!! Thanks DH, Good luck everyone else!

I’m back! Thanks.

Just got my mail. Looks good.

March 26th, 2013 at 7:11 pmsend him back home Says:

CptNerd wins the internet tonight!

March 26th, 2013 at 7:11 pmJames Sommer Says:

On the Internet, we can all be James Sommer! Woot!

Oh my goodness you guys, please get it together!

I had been monitoring uptime with pingdom, and the last month’s 99.80% uptime was already concerning.

Now with this mega outage, we are down to 99.68% for the last month, and dropping still…

Is there any accountability here, other than us taking our business elsewhere? You advertise 100% uptime, can we at least get money back?

The only reason this doesn’t loose gigantic business for you is because it’s a pain in the butt for us to switch providers. That is not a good reason to be holding on to your customers, DH…

sweet, back online :)

March 26th, 2013 at 7:11 pmAnomolous activity Says:

I was ruled out :(

@James Sommer — No, unfortunately. Someone just filled in the same information as you did (name and URL).

Looks like things are back…

….for now.

I’d love to see a custom error page indicating downtime on any effected sites whenever these types of issues come up; therefore, our members / customers know what is happening instead of the dreaded, page not found…

My email is back up… thank you Dreamhost for all your hard work. And to those people who posted using my name, well I guess that was my fault for using it in the first place. I hope it gave you some enjoyment to do that.

Thank you once again Dreamhost.

Take care,
Me

What’s going on Dreamhost?
Hung

god bless me, it’s back

e-mail is back!

my site and mail is down. any help?!

My sites are up now.

The waahmbulances are being called back to the station.

Just curious, I’m in the DC metro area, any reason why my server is in LA? I assume it’s a capacity issue, and it’s not really that important, but I alway like learning about the network topographies I use…

email back up!

All of my websited are down :(
As well as dreamhost.com itself…

DanielT, believe me, no one missed your website during the outage.

Absolutely, positively no one.

Thanks for handling it as quickly as you could do. Frustrating when it happens!

Great work guys..my sites are back up! All you guys looking to jump to the irvine servers might like to know we were down as well. Tip of the hat to the DH techies!

@Righteous, you need to add dripping ooze and an ominous “DAH daaaaaaaah” music to your msg….

Dreamhost panel is working – via the IP address. What’s funny is that it’s not even a Dreamhost IP – they’re using Softlayer servers to run the panel! Hmm…

I’M BACK! /orgasm

Macc, your site is up. Neat game.

“At this point, we are suspecting our Cisco router to be the cause of the incident.”

No wonder Cisco’s stock took a dump today.

March 26th, 2013 at 7:17 pmKrystal Baul Says:

Prepare for more downtime in a few hours.

flip the green switch to on, turn the left knob labeled VGQ to 4.5, and take the red pill .. websites are up

Can you let us know if any email will be lost or will it just be delayed?

“our cisco router” sounds odd. Only one hosting location? Only one router? Only one internet connection, even?

Oh the drama, I wonder how many lives, marriages, mortgages and businesses were lost during this relatively short outage. Oh well, on with my Kirtan music class I go.

FYI my sites never went down, control panel went down but is working now (albeit slowly).
My email seems to be working but did not receive a test email I sent to myself — with no bounceback yet either. Weird

Everything looks to be back up now. Sent a Credit ticket through Support. We’ll see what happens. :)

@Dave ..who needs mail use the phone @gt .. nope use two cans and a string, make sure the knot is tight in your can

Thanks for fixing it so quickly, as well as maintaining a good uptime for a very low cost.
Ignore all the crybabies.

@dlocutor: tighter knots helped. Thanks!

March 26th, 2013 at 7:25 pmelliot smith Says:

@Dave the answer to your concern as always here at DH downtime services is that during the outage all incoming e-mail is F— Y–
We at DH trust this answers all concerns you might have.

what’s fuck ????

Again?????

DREAMhost == NIGHTMAREhost

Down again? my sites are down again…

@flavio and @Alex T — I don’t see any downtime. What exactly are you two seeing that isn’t loading?

I got a letter in January telling me my sites would be migrated to Ashburn, Virginia. How come my sites were down if the problem was in the LA datacentre?

I don’t know what everyone was complaining about. My sites were down for about 2 hours at most. For those of you hosting business critical sites on here, you all need to look into VPS and redundancy. Get a second, low-cost host and use a DNS failover service so that if your site goes down it will redirect your visitors to your alternate site. Downtime is a part of life in this day and age. Yes it sucks, but it could be worse. I’ve been with DH for many years and I have to say that I almost NEVER have downtime. As webmasters, you need to take it upon yourself to practice redundancy. DH has been doing a great job at making their systems more redundant, but the first point of failure will always be having only 1 host for business critical systems. Do your homework, put on your big boy/girl pants and ensure your systems are redundant if you simply cannot afford downtime. It’s that simple. No need to go crazy on the board or result to foul language like some degenerate, just be proactive.

Tnx for fix this. From Argentina.

@SongCloud – you can’t do failover with round-robin DNS.

Thankyou thankyou thankyou for the quick resolution

yikes…bleeding dollars here!

@Elizabeth – I am in Brazil, as example my site http://www.unys.com.br is not open, and no on other in server TIFT

I’m guessing it is taking a while to get all the sites back up and running. I have a site offering video streaming which was working properly before this issue. I will wait a little longer to see if things change…

@Alex T – Server tift is not part of this issue. It’s being worked on right now.

Come on! I still getting a 500 error!

@Elizabeth – I need to open a ticket support? or are you team already fixing the problem?

@Alex T – You can open a ticket if you like, but the server is already being worked on.

–2013-03-26 19:33:10– http://www.mafiadolixo.com/
Resolving http://www.mafiadolixo.com... 108.162.199.236, 108.162.198.236
Connecting to http://www.mafiadolixo.com|108.162.199.236|:80… connected.
HTTP request sent, awaiting response… 502 Bad Gateway
2013-03-26 19:33:14 ERROR 502: Bad Gateway.

Scary that this outage took down your homepage. I mean, it’s the first place customers go when there’s a problem, right?

My sites back up. Thanks for fixing it so quick. Hardware failures suck, but they happen. Remember the triple generator backup failure that had sites down for about 6-8 hours a few years. Now that was bad. ;-)

are you guys serious?

Are Justin and me the last customers for DH? Anyone else down?

Like a heart attack!

really?

Down here too, not that it helps much to post that info here.

And no official word from DH yet….

Back up at the moment. Hoping it lasts aside from some rolling outages for the migration. This has been an incredibly stressful two days answering clients with unfortunately negative replies due to outages. This has to be the end of these frequent outages, or I will sadly have to move all of my clients sites and my business off of DreamHost.

We’re working on keeping things online, and are moving forward with our emergency maintenance, which should permanently resolve these issues. Thanks for your patience guys, we know how trying unplanned outages like this are.

That’s why now I only use dreamhost as my “test garbage” thing

I’m still down….

Great. I was in the middle of some work on one of my sites and now it is gone.

3rd outage?

Up now….

@Roland
@Michael

We’ll continue to see some up an down behavior as we move forward in our maintenance. Please use the maintenance post linked in the main OP for any further comments, as we’ll be closing this post and moving the status updates and comments to that post.

i was on chat support earlier today and the tech was trying to tell me my wordpress sites needed to be optimized.

what a bunch of crap!

if you can’t ping it, ssh, and the vps is running at a very low load an using 1/3 of the allocated memory, what does wordpress optimization have to do with down time.

nightmare host!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Power disruption affecting US-West Data Center (Irvine, CA)(RESOLVED)
3:01 pm

Power disruption affecting US-West Data Center (Irvine, CA)(RESOLVED)

Posted (March 19th, 2013 at 3:01 pm PST) by andrea

Emergency maintenance for normal power operations transfer. March 20th 9pm to March 21st 1am.

As of now the transfer back to UPS power and normal Irvine DC power operations is scheduled to start at 10pm. We are opening a 4 hour window for this emergency maintenance. We are pleased with the results of the fixes implemented and the testing that has thus far coincided. We do not anticipate this maintenance to cause another power incident. We will update this status post as we progress through this emergency maintenance.

Update March 20th 11:00pm PDT: Our data center providers emergency maintenance is complete and we are back on city power. This means we are operating under normal conditions with all redundancies back in place. We appreciate your time and patience throughout the issues we’ve experienced over the past day and a half. As always, if you have any questions or concerns, please contact tech support.

Update March 20th 9:40pm PDT: We are now on utility power and off of the generators. We are monitoring the situation and pleased to announce we have seen no impact. We will continue to monitor until we are 100% we’re in the clear.

Update March 20th 9:10pm PDT: Cut over is about to happen right now. Our team is ready to react if there is any impact. We are not expecting any, but we are prepared if anything happens.

Update March 20th 8:45pm PDT: In the spirit of getting things resolved as soon as possible, we are moving the support window to 9pm as our engineering consultant has been on site for some time now and is comfortable with the proposed changes.

Update on Irvine Data Center Power Systems Status and Upgrades – March 20th 7:00pm PDT

All DreamHost systems and services in the Irvine DC are up and stable, and have been since early this afternoon PDT. The Irvine DC continues to run on generator power, and there is redundancy in the generator setup with only two of four generators being used right now. The other two are ready for automatic failover in the event of any issues with one of the running generators. These are large, 600KW output generators, and two of them have more than 3x the capacity to power the current systems in production in the Irvine DC.

UPS systems update: Alchemy has had two expert power engineering teams in troubleshooting their UPS system, and they have identified through testing that a core mechanical circuit breaker in the UPS system was the point of failure in both power outages. A contractor Emerson delivered a new replacement part to the Irvine DC around 3pm PDT today, and procedures to replace the circuit breaker were developed by Alchemy engineers and reviewed by our team over the next hour or so. At 4pm PDT our team (including me, our VP DC Ops, and our own power engineering consultant) discussed the part install and testing procedures, and gave the green light to install the part. That was completed around 5.30pm PDT, and testing of the UPS system under load commenced at 6pm PDT. The testing is very low risk, and is currently being monitored by our Irvine DC ops team.

We expect full testing of the UPS system – including the newly installed circuit breaker – to be complete by 8.30pm PDT. If the test results are all clear, we will do a final review of the Alchemy plans to transfer critical power load from the generators back to the UPS system, and then open an emergency maintenance window at 9pm PDT for Alchemy to complete the transfer back to UPS power and normal Irvine DC power operations. The planning of the transfer has been done to minimize any risk of power issues during the transfer.

We will post another update after the testing is complete, estimated to be around 8.30pm PDT.

Update from our CEO on the March 19/20, 2013 power outages affecting services in our US-West (Irvine, CA) Data Center

I would like to share more details with our customers concerning the power outages and resulting network and systems issues that impacted our services on March 19/20 in our US-West (Irvine, CA) Data Center (the “Irvine DC” for short).

A third party, Alchemy Communications, manages the Irvine DC. We lease a large secure space in their facility. Alchemy is responsible for providing power, cooling, security and related infrastructure services and maintenance. The facility has a good track record on all of these responsibilities – including providing reliable power. However, yesterday at approximately 3pm Pacific Daylight Time (PDT), there was a failure in their Uninterruptible Power Supply (UPS) system that completely shut down power to all network and systems housed in the Irvine DC. This power outage affected all tenants and was not limited to just DreamHost’s equipment.

The power systems at the Irvine DC are designed to be redundant. The UPS system is “in-line” and in the event the power grid feeds go down, the UPS provides power until the diesel-powered generators kick in. We believe, at this time, that Alchemy was performing unannounced maintenance on their UPS systems and the systems failed – resulting in a complete power outage. In addition to their UPS systems failing, their generators did not kick in.

The power failure lasted just a few minutes, however it created a number of major issues with our network and systems in the Irvine DC that took many hours for our operations teams to recover from. Not the least of which was the loss of several critical pieces of networking hardware which did not survive the power event. Complete details of these issues will be shared once we have completed a detailed review over the next couple of days.

All customer-facing systems were largely restored from the first power outage by early this morning March 20, 2013 PDT.

After the first power outage, we were assured by Alchemy that their power systems would not be worked on further until a detailed, tested plan was in place that would guarantee no additional loss of power. However, at sometime around 4:30am PDT today their UPS system failed for a second time. This resulted in another complete power outage and another intense period of reboots, restores and system checks from our team. The time to restore most services in the wake of this second power outage was much quicker, mainly because there were no resulting hardware failures and we had learned from the first failure. Alchemy has opted to run the Irvine DC on generators until the UPS issues are fully identified and resolved, and we are monitoring the situation closely to do our best to ensure that there are no further power outages or issues that will affect our services in the Irvine DC.

Corrective Action: Alchemy has a team of UPS specialists and Edison power engineers on site who have identified potential points of failure in the existing UPS infrastructure. They are currently in the planning phases of a repair to the UPS systems. The proposed power system enhancements will be subject to rigorous review and testing before being implemented. If all goes as planned, the facility will be able to switch back from generators to grid/UPS power. At this time, we do not believe that a public grid power failure contributed to either of these incidents.

DreamHost will have additional network, systems and data center engineers assigned to monitor all systems in the Irvine DC during the UPS system upgrade. We also have oversight of the proposed UPS system repair plan being structured by Alchemy. We will post an update on dreamhoststatus.com as soon as the timing of any planned power system maintenance is known. We are working diligently to ensure that the planning and implementation of any UPS upgrades, maintenance and/or the cut back to grid/UPS power will not affect continuous power to our systems and will not impact our customers.

Last, but not least, I want to apologize for these service disruptions. I know how critical our services are to our customers and their livelihood. I fully recognize that any disruption to services can affect important production environments and projects. Our team will work diligently to ensure that we mitigate the power issues going forward, including a full audit of all facilities that house DreamHost customer data. We will learn from this event and continuously improve our operations and services.

All customers impacted by these service issues can apply for credits or refunds in accordance with our guaranteed uptime policy by contacting support through the panel.

Sincerely,

Simon Anderson
CEO, DreamHost

Original post:
We are currently experiencing power disruption in our US-West Data Center (Irvine, CA). Our system engineering and data center operations teams are looking into the cause. We will update this post as soon as we have more information. We apologize for the inconvenience this is causing.

Update March 20th 2:40pm: We are confident that the DreamObjects service is now 100% functional again. There was no data loss due to the power disruption. We are keeping this post unresolved until we are off generator power. Please see the update from our CEO above for more details.

Update March 20th 1:30pm: We are nearing the end of this disappointing and unfortunately long issue at our US-West Data Center in Irvine. We will continue to provide updates until we are 100% confident that we are not seeing any further impact from the Data Center power outage. Please bear in mind that we are currently running on the facility’s backup generators. This means that there will be a cut over to building power once the Data Center is confident that the UPS maintenance they are performing is successfully completed. We already have a plan in place to make sure that change goes as smoothly as possible. Our CEO Simon Anderson will also be updating this status post today with a detailed account of what happened and what our plans are moving forward after this incident.

Update March 20th 12:30pm: Almost all Dreamhost Objects issues are now resolved. There is a small percentage of groups that may be seeing 504 errors however, we are still working to resolve these last remaining issues. There is no data loss whatsoever for any DHO accounts. All other services related to shared, VPS and dedicated hosting are working and have been resolved for some time. If you are still experience any issues with your service, it is no longer related to this power outage.  Please contact support so that we can assist you. Updates to this post will now be spread out to one hour intervals while we continue to restore DHO services. We understand your frustrations and thank you for your continued patience.

Update March 20th 11:20am: Dreamhost Objects is now serving content, but we are still working to resolve some lingering issues   All other services related to shared, VPS and dedicated hosting are working and have been resolved for some time. If you are still experience any issues with your service, it is no longer related to this power outage.  Please contact support so that we can assist you.  Updates to this post will now be spread out to one hour intervals while we continue to restore DHO services. We understand your frustrations and thank you for your continued patience.

Update March 20th 10:10am: We are continuing our efforts to resolve the issues with Dreamhost Objects.  All other services related to shared, VPS and dedicated hosting are working and have been resolved for some time. If you are still experience any issues with your service, it is no longer related to this power outage.  Please contact support so that we can assist you.  Updates to this post will now be spread out to one hour intervals while we continue to restore DHO services. We understand your frustrations and thank you for your continued patience. 

Update March 20th 9:30am: Dreamhost Objects services are still being worked on.  Some VPS host machines are being rebooted to clear up some lingering issues from earlier this morning. If you are on shared hosting or have a dedicated server and are still experience any issues, it is no longer related to this power outage.  Please contact support so that we can assist you.  We understand your frustrations and thank you for your continued patience.

Update March 20th 9:00am: Dreamhost Objects services are still being worked on.  If you are still experiencing problems then your issue is no longer related to this power outage, with the exception of Dreamhost Objects customers.  Please contact support so that we can assist you.  We understand your frustrations and thank you for your continued patience.

Update March 20th 8:35am: Dreamhost Objects services are close to be fully restored.  All other services are restored from these two outages.  If you are still experiencing problems then your issue is no longer related to this power outage, with the exception of Dreamhost Objects customers.  Please contact support so that we can assist you.  We understand your frustrations and thank you for your continued patience.

Update March 20th 8:00am: Dreamhost Objects is making progress towards resolution.  All other services are restored from these two outages.  If you are still experiencing problems then your issue is no longer related to this power outage, with the exception of DreamObjects customers.  Please contact support so that we can assist you.  We understand your frustrations and thank you for your continued patience.

Update March 20th 7:30am: All services have been restored with the exception of Dreamhost Objects. We are  continuing to troubleshoot those few remaining issues with Dreamhost Objects and should have it back online shortly. We will continue to update this post and once DHO is back online, we will resolve this outage completely.

Thanks for your patience!

Update March 20th 6:55am: We are still  continuing our efforts  to get Dreamhost Objects back online. Most of our other services have been restored such as shared and dedicated hosting and we are seeing the number of service issues drop. We will continue working until we are confident all services are restored.

Update March 20th 6:35am: Our entire team is continuing their efforts in getting all servers and services back online. Many of these services have been restored, such as shared hosting and we are seeing the number of service issues drop. We are still addressing dedicated servers, and they have started to come online. We estimate 99% of shared hosting, 95% of VPS and 99% of dedicated to be restored at this time. We are working to get all of your services restored as soon as possible.

Update March 20th 5:55am: Servers and services are coming back online as fast as we can possibly get them up. We do need to tend to certain machines that do not come back on reboot, but so far we are able to get service restored. The majority of our team has been on this issue since the first data center power outage, so we are pleased we are able to restore service much faster. Dedicated machines are next on our list to get back up. We have already started on those so they should be coming up as we move forward.

Update March 20th 5:15am: We are in a much better status than we were with the initial outage.  We are able to tend to machines individually to get service restored.  We are seeing services being restored slowly but surely, though it is a bit of a process.

Update March 20th 5:00am: Power has been restored once again, and we are currently running on our backup generators as we work to bring services back online. We understand the frustrations this is causing, as we are in the same boat as you. Please hang in there, and support us as we work to get everything up from this second data center power outage. We sincerely apologize for the inconvenience.

Update March 20th 4:30am: We are experiencing another power disruption in our Irvine datacenter. Our teams are currently investigating; further details to come.

Update March 20th 3:50am: It looks like all services affected by the outage should be restored by now. If you are still experiencing downtime, please contact support, as it is most likely not related to this issue. Thanks once again for your patience while we’ve worked to resolve this!

Update March 20th 3:00am: We are still clearing up a few of the remaining issues that were caused as a result of the outage, and continue to grow closer to resolving this completely. Fortunately, most services and sites should be up and running by now. We will post a final update once we’re fully back online.

Update March 20th 2:16am: Most sites and services should be back up now, save for a few remaining issues we are still working on clearing up. Thanks for your patience!

Update March 20th 1:35am: The rack we were experiencing issues with in the previous update have been resolved, and services should be back up in the next few moments. We are continuing to work toward a final resolution toward this outage, and will update again shortly.

Update March 20th 12:19am: Services are still slowly but surely coming online. We’ve had one rack of machines giving us a particularly tough time, and are working on that now. We should have more fixed in the next few moments.

Update March 19th 11:41pm: Services are slowly coming up, and addressing each one as we get the next fixed. Also, our data center has extended the UPS maintenance window to 3am PDT.

Update March 19th 10:56pm: Services are still coming up, however, some hosts require manual intervention. We’re seeing some sporadic outages as well, and are working to get those fixed ASAP!

Update March 19th 9:35pm: Ok, we see a lot more sites and servers coming online, so this is good news! We’re still monitoring things, to ensure they come back online smoothly.

Update March 19th 9:15pm: We’re still working away on this, and are now seeing some services restored. You may see some intermittent outages for a bit while we work out the kinks, but services are coming back online slowly.

Update March 19th 8:30pm: We’re having some success resurrecting part of the problem areas of our network. This does not mean hosts are up just yet, but it does mean we’re getting closer.

Update March 19th 7:55pm We are still getting the new network device in place and configured. We’re doing everything we can to work as fast as possible and restore service ASAP.

Update March 19th 7:19pm: Our data center provider is also initiating emergency maintenance on the UPS system that caused the initial power outage. This maintenance is scheduled to be complete at 11:55pm pacific time. We don’t anticipate problems to stem from this, but are bracing for issues, should they pop up.

Update March 19th 6:30pm: As of now we have two network devices, that handle traffic internal to the data center, that are down. One main one, and one backup. Our vendor has said, that BOTH devices were fried during the power outage, and are suggesting we RMA BOTH of them. We are in the process of deploying a spare to that location and we estimate that will take 1-2 hours. We will continue to update as we get more information.

Update March 19th 5:55pm: We are still working on this problem, however it’s proving fairly difficult to resolve. We do have power in our datacenter and most of it is up, however the lingering issue is a routing table we’ve been unable to restore. We’ll continue to update here.

Update March 19th 5:34pm: At this time, we’re still working on the problem, and are going through the many troubleshooting steps needed, however these steps take a bit of time to complete. We’re working as fast as possible to get things sorted out ASAP.

Update March 19th 5:10pm: While we’re continuing to work on bringing all the services back online, we’re happy to report that about 90% of DreamObjects are now available again, and we confirmed that there is no data loss. We appreciate your continued patience while we are working on the rest.

Update March 19th 4:58pm: Despite the diligent work from our technical teams, we are still not fully back up with our services in this location. We are very sorry, and will update this post again with any news we may receive.

Update March 19th 4:28pm: I’m afraid we are still not fully back up at the moment but continue working on it. We are very sorry for the inconvenience this delay is causing and appreciate your patience.

Update March 19th 4:08pm: We are slowly but surely bringing all services back online, including DreamObjects. We are very sorry for the inconvenience this delay is causing.

Update March 19th 3:47pm: We are still working on bringing back all affected services. You can check if you are affected by this power disruption by looking up where your services are hosted in the DreamHost Web Panel. Please note, DreamObjects are also located in this data center. Again, we apologize for the inconvenience.

Update March 19th 3:27pm: After the power came back up, we were able to bring back part of our services. We are slowly but surely bringing all of them back up, so your sites may take a little while longer to start working again. We appreciate your patience during this process.

Update March 19th 3:07pm: We now have power back in the data center. We are investigating the cause, as it affected the whole facility and not just us. We are also working on making sure that all services are coming back online as expected. Your sites may take a little while longer to come back up.
You can check if you are affected by this power disruption by looking up where your services are hosted in the DreamHost Web Panel. Please note, DreamObjects are also located in this data center. Again, we apologize for the inconvenience.

This entry was posted on Tuesday, March 19th, 2013 at 3:01 pm and is filed under General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

1,539 Responses to “Power disruption affecting US-West Data Center (Irvine, CA)(RESOLVED)”

Well, were down :(

Down here. le sigh

Losing. Money…

I can not believe i am paying $60+ a month for this service!!! One word described you guys – #appalling!

Your continuous outages is a major reason why i no longer refer clients to DH.

Yah, just wonderful. What an awesome place to host an e-commerce service! Redundancy would be so… redundant.

I hope you will fix it shortly! Good luck and thank you for your service!

Isn’t this the data center they just finished doing stuff to yesterday?

It would be nice if they put in some sort of redundancy to avoid this sort of crap in the future.

Sucking up doesn’t work either.

And you would think a web hosting company like dreamhost would have some kind of redundancy!

Oh gosh… its so primary…. energy problems.. for..a… DATACENTER! omg..

I assume that must be why my dedi (DS2032) is down? Also I have been getting a lot of downtime reports from a third-party reporting service I use so maybe there are more issues that need to be looked at. I know this stuff happens from time to time. However, you say 100% uptime, I suggest you change that since is not exactly true…

Hope you guys can get it all ironed out soon!

Our site is completely down. Please advise when you expect to have this back up and running.

“”””Also I have been getting a lot of downtime reports from a third-party reporting service I use so maybe there are more issues that need to be looked at. I know this stuff happens from time to time. However, you say 100% uptime, I suggest you change that since is not exactly true…”””””

I’ve been having the same issues.

I´m DOWNNNNNNNNN

DreamObjects = down :(

Second day, monday with best traffic, and today. No traffic, no money. The Clients are complaining.

Do I need to Migrate to Amazon Cloud!!!???

All mine are down as well.

even shittyDaddy have better uptime than this crap

I SPENT BY MONTH 5.000 DOLLARS IN MARKETING, 99$ TO DREAMHOST, AND YOU HAVE COURAGE OF DOWN MY E-COMMERCE. YOU ARE A BIG CRAP!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Just the other day I was lamenting on how few issues I’ve had with DH over the past year, especially considering some of the stuff that has happened here over the past 2 years. I guess it’s nice to know it’s not on my end…

I’m curious how it is that current dreamhost time is 15:00:00 hours but this post was made at 15:01. Are some of the times on this page being translated into my time zone? If so, why make the point of displaying Dreamhost time all over the place?

Just a suggestion. I hope you will find a way to have some redundancy in the future. And I wish there were some way I could get my clients’ data out of the god-awful Irvine data center and into one of your other locations. That’s the one that was having all the problems a few months back.

Same over here, all my websites down, clienst are complaining

losing more in this hour than all of hosting cost every single month. might finally be worth it to hire a sys admin and move off this unreliable shit…

Fix this fast plz! A energy problem is unacceptable in a 100% uptime Datacenter!

March 19th, 2013 at 3:09 pmWisdomWouldBeWelcome Says:

http://blog.datacentermapping.com/2012/03/01/providing-reliable-data-center-backup-power/

Just tune out the sponsor name, and shows what kinds of things a serious hoster does to avoid these kinds of problems.

Of course, maybe the problem is ‘within the ring’, a fine place for faults to be present after all supposed measures are in place.

Just ask Boeing, about their Dreamliner (!) batteries…

Clearly only affecting newer servers, as all my accounts set up more than a couple of years ago are working fine.

Do you complainers even read before posting? It says the staff does NOT read these comments, if you want to talk to the staff about it file a support ticket. And if you’re running a site where any downtime results in lost revenue, you shouldn’t even be using a cheap service like Dreamhost. You should be on a more expensive service that offers redundancy with multiple data centers across the country that take over if there’s downtime.

We are paying for a data operation, where is the redundancy? How can there be so many power outages???

Down again. $120 for a dedicated service that goes down at least twice a day without fail.

Do not host client sites on Dreamhost. I learned that a long time ago, and I’ve been with Dreamhost for probably 6 years.

March 19th, 2013 at 3:11 pmTrinity Sales Says:

This is ridiculous man my websites is down…

Yes, we know they don’t read these (or don’t admit to), but it helps make us feel better to vent and see others in the same boat.

Also, Micke, that doesn’t seem like the most intelligent way to spend that $5100. Try 4k on marketing and put that extra 1k into the system that runs your business.

We are down. San Diego, CA

Very interesting. I run in a couple data centers for many years and once have I had a power outage and that was an unexcused mistake in power cycle connections. Very surprising that my data centers in Greece have a much better setup…

Back-up generators?

Downnnnnnn PLz redundant energy sourcesss!!!!

We are paying for 100% uptime.. this is not 100% uptime. Our site down is costing me money every single time it goes down.

Only been with dreamhost for 6 months, the reliabilty has been terrible

Dreamhost should change name to Sleephost!

My site is down. This happens too often. Please work on making your hosting more reliable. Thank you.

I have to laugh at the dreamhost guarantee
Enterprise-Level Reliability
DreamHost manages its own network, so that your site never skips a beat. DreamHost’s network is backed by a 100% uptime guarantee policy and you’ll have around-the-clock (24×7) access to our expert technical support representatives via email, and live chat

— gotta go angry customer calling!

Hey Dreamhost. Ever heard of batteries or generators?

OMG! Robots Forex also down. Hope for sooner fix! DH the best anyway!

Honest question. How hard is it to change web hosting companies???

All I can do is laugh at it, stop whatever I was doing and “take a break”. This only happens at Dreamhost nowadays.

Again? site down!!

March 19th, 2013 at 3:19 pmMonty Brewster Says:

“@PL: Dreamhost should change name to Sleephost!”

I have no words

VPS – MySQL still down!

The power may be back on, but none of my sites are back up.

How my fuckking god a external non related facility problem can down energy in all datacenter? Explain yourself plz. Where are the generators and batteries???

so the we’re guarnateed the right to complain to the staff? Hah – I’m going to try that with my development “We guarantee you the right to bitch about our service to our staff if you don’t like it but you still need to pay the bill.”

Classic DreamHost… I’ll not accept this again, bye!

Hurts to just launch first blog post and then promote it and then it links to blank page…ouch, please fix

So they say they have power back but my sites are still down.

I have been with DH for about 10 years. They are better than others I have been with including Canaca, HostGator and GoDaddy. I understand they have downtime but they need to scrap the 100% update thing because its false. However, I have seen way worse that DH. I was with LVCS and their data center burnt down to the ground and they wanted the effected customers to move to a more expensive plan as a solution. Simon needs to just go on a changing spree… scrap the 100% uptime thing because its impossible, get some sort of backup plan and mitigate these conflicts from happening.

March 19th, 2013 at 3:21 pmAnderson Pereira Says:

tomorrow is the inauguration of my site… PLEASE… fix it!!!
i have to upload the files!

well, we are still down here

I´m down

Satisfaction Guaranteed …

hay que ser realistas, es mentira que me voy a otra compañia de alojamiento web. Tengo muchas páginas alojadas aca y estoy muy comodo con el uso del panel de control y todo.

Pero cada vez baja mi satisfacción, cuando encuentre un mejor alojamiento que sea serio, no lo voy a dudar y me voy de un vez.

I’ve been with Dreamhost for over a year. My sites have only been down a few times, and even then, it wasn’t that long. Thanks for keeping us updated!

Stilllllllllllll dowwwwwwwwwwwnnnnnnnnnnnnnnnnnnnnnn

We now have power back in the data center. We are working with the facility provider to investigate the cause, as it affected the whole place and not just us. We are also working on making sure that all services are coming back online as expected. Your sites may take a little while longer to come back up.
You can verify if you are affected by this power disruption by looking up where your services are hosted in the DreamHost Web Panel under Status/Data Center. Please note, DreamObjects are also located in this data center. Again, we apologize for the inconvenience.

AGAIN!? Every month I’m getting at least 2 episodes of downtime with long duration. 100% uptime?? Are you serious??

I left a hosting unreliable for another worse! : (

Still down here, is anyone back up yet?

March 19th, 2013 at 3:24 pmDreamy Hoaster Says:

I love coming on here and seeing everyone whine bout their Web site being down. FIRST WORLD PROBLEMS, LOL!

I am a relatively new customer and I will definately be changing service providers. I believe an ISP should have backups, otherwise we would host the site in our own server room. It would be nice to be able to speak to a human on a telephone.

March 19th, 2013 at 3:25 pmMonty Brewster Says:

I’m in the UK, its 22:19 here…. couldnt give a fuck if this happens a this time, you’ve got hours lads take your time! lol

Still down

This is getting ridiculous. DH is getting really bad. Final straw…cya.

March 19th, 2013 at 3:25 pmWisdomWouldBeWelcome Says:

Ok, looks like the power loss was on one critical rack — and may be the area where they are completing the big network upgrade that has been going on for months.

Thus, quite understandable, if it is sure to raise hackles.

Since this big upgrade is supposed to be completed with switchovers scheduled on Wednesday (tomorrow), hopefully we’ll see one more per-site short knock-down, and from then on things will be…wonderful.

I hope so ;) — and commiserate the folks in the center of the storm at DreamHost!

to Anthony…its spelt appauling not appalling!

I just got on the computer to update my site and server goes down. Anyone know what was the cause of the power outage?

down here too…

How much time does it take for a server to be online again? 10 minutes? 15?

March 19th, 2013 at 3:26 pmMonty Brewster Says:

“John Says:
I am a relatively new customer and I will definately be changing service providers. I believe an ISP should have backups, otherwise we would host the site in our own server room. It would be nice to be able to speak to a human on a telephone.”

— Aw, somebody phone poor lonely John, he needs a fwennnndddd!

My hosted WP blogs are still down, despite the restored power :-(

to those people getting all upset – its a power outage of the whole centre…relax it’s not the end of the world!!!

March 19th, 2013 at 3:27 pmWisdomWouldBeWelcome Says:

Meant to say, thus the problem was indeed ‘within the circle’, like the Boeing one.

I couldn’t find info on this US-West datacenter, but I did on a couple of other Los Angeles area Dreamhost colocations, and they appear to have a sound set of power and network redundancies, on a very short look.

Are any of your sites up yet, mine is still down. Losing money and not happy, should have known better Dreamhost…you got me. 100% UPTIME=LIE

Nope. Not up.

March 19th, 2013 at 3:29 pmMonty Brewster Says:

“Your sites may take a little longer to come back up”

READ THE FUCKING POST FFS

Have a presentation in 10 minutes… any chance it’ll be up by then? :(

Our site is down too. Who let Lindsay Lohan into the data center again? You guys said you would stop letting her into the data center, and telling the court that it was a rehab center.

Either teach Lindsay linux, or get rid of her. I’m sick of her puking on my VPS.

Post @ 15.07 saying power is back. its 15.25 and sites are still down.

first world problems…some of you people make me sick…at least you have money to lose…unlike millions of other people out there…

100% uptime ????

Bad timing. :( I JUST started an ad campaign and just got featured in a media outlet today. Crap and phooey. :(

I just wonder how many of you commenting are competitor hosting services trying to harm the DH brand… :-\

March 19th, 2013 at 3:31 pmMonty Brewster Says:

@No No

No

Man, that Irvine data center is like the Enterprise’s port nacelle. It always seems to be the one to catch the most shit.

@Monty Brewster
no, what?

March 19th, 2013 at 3:32 pmMonty Brewster Says:

“Jamal Says:
Man, that Irvine data center is like the Enterprise’s port nacelle. It always seems to be the one to catch the most shit.”

– Best post on here EVER

March 19th, 2013 at 3:32 pmWisdomWouldBeWelcome Says:

Jamal, that’s why they’ve been fixing it up with new network services. Read the notifications?

For those giving us crap about first world problems, some of us live paycheck to paycheck and we need the income from our websites to pay the rent. This IS a big deal for some of us.

Con someone please get the hamsters running in the wheel again

Arrrghhhh i’m down !!!!

I’d say they definitely read these sections, Their competition does for sure. They posted the problem, so I’d rather them be fixing it than responding here. Time a tickin’ though.

30 mins down…

March 19th, 2013 at 3:33 pmMonty Brewster Says:

@no no

Just no, you’re wrong, so no. You no?

Hah ha! SG, are you serious? “I just wonder how many of you commenting are competitor hosting services trying to harm the DH brand… :-\” Dreamhost needs no help in this regard, and your comment is quite naive. Do you perchance, work for Dreamhost yourself? That sounds much more likely.

Our website is still down.

March 19th, 2013 at 3:34 pmMonty Brewster Says:

“ADMR Says:
30 mins down…”

– This guy has a WATCH!!! Get him!!!

stop blaming Dreamhost for everything… if you were decent developers, then u would have your own disaster recovery plan in place for times like these…

@Monty Brewster

Update March 19th 3:07pm: We now have power back in the data center.

Current DreamHost time: 15:29:21 Pacific

March 19th, 2013 at 3:34 pmMonty Brewster Says:

Kevin Doyle just used “perchance” WINNING

March 19th, 2013 at 3:35 pmMonty Brewster Says:

@No No

and the bit after that…….. you know the bit that says something about it may take some time for your sites to come up? hmmmm?

Most of us are NOT developers. We are just small-business people. Does that mean we should not expect reliable hosting?

30 minutes down, no sign of return for me. That’s a big black mark for a dedicated server at this price, quite disappointed and hope it’s restored quickly.

CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!

I recently switched from GoDaddy for my hosting and domain management. In the 10 years that I’ve used them the only time that I can recall an outage was just before I signed up for DH in September of 2012 (this outage at GoDaddy was part of the reason I switched to DH and only affected a couple of sites that I manage). I am paying more for my service at DH and I have downtime within 6 months? That is unacceptable, especially for something as simple and negatable as a power problem.

We were featured in a New York Times article (on the front page today), and this is a very bad time for the servers to go out… when we’re getting most of our traffic. There will be a lot of potential lost.

@Kevin Doyle

No, I don’t work for DH. I just read all the trolling here. Too bad I wasn’t in the soap business, a lot of people could use a bar stuck in their mouth here. ;)

Hope you get it fixed soon. For those who are moaning, Dreamhost can hardly do anything about a Powercut – it’s just one of those things.

Even Facebook goes down sometimes, it can’t be helped. So those moaning, put yourself in Dreamhost’s shoes and shut up!

Regards,
Rob

I feel you, Ashley. I moved everything here a few months ago and I have had a few outages here already. :(

I was wondering if you can send out warnings to my email address when the server is down, and when the server eventually gets back on, like the one we are experiencing currently. The reason being is that I am using Google Adwords and when the site is down, I will get charged a lot of money for an obsolete and blank website. Once you email me that the server is down, I will be able to pause my Google Adwords account and then initiate it once the sever is ready. The amount of money I’m talking about is approximately $20-$50 per click.

Thank you for your time,

Neil

Well my sites have been down for about 3 hours now.

Well we’ve been down for about 30-45min now, paying well over 100 a month. Not Good! Not Good!

el día que dreamhost regrese el dinero a todos sus clientes por el tiempo caido, ese día deja de pasar esto.

March 19th, 2013 at 3:39 pmMonty Brewster Says:

Now this is the story all about how
My life got flipped, turned upside down
And I’d like to take a minute just sit right there
I’ll tell you how I became the prince of a town called Bel-air

In west Philadelphia born and raised
On the playground where I spent most of my days
Chilling out, maxing, relaxing all cool
And all shooting some b-ball outside of the school
When a couple of guys, they were up to no good
Started making trouble in my neighbourhood
I got in one little fight and my mom got scared
And said “You’re moving with your auntie and uncle in Bel-air”

I whistled for a cab and when it came near the
License plate said “fresh” and had a dice in the mirror
If anything I could say that this cab was rare
But I thought nah, forget it, yo homes to Bel-air!

I pulled up to a house about seven or eight
And I yelled to the cabby “Yo, homes smell you later!”
Looked at my kingdom I was finally there
To sit on my throne as the prince of Bel-air

We were in the process of transferring data, hopefully our database will not be corrupted!

All I can do is laugh at it, stop whatever I was doing and “take a break”. This only happens at Dreamhost nowadays..

Neilytron – it’s a paid service, but try https://www.1stwarning.com. it’s what I use to let me know when DH goes down. Sad that I gotta pay to know when I’m not making money…

March 19th, 2013 at 3:42 pmMonty Brewster Says:

Who wants to play tictactoe?

This is really outrageous how often you guys are down. If I wasn’t technically incompetent I would have changed hosts long ago. Dreamhost is terrible.

Wow, they finally get traceroute working again, just in time to show you can’t get there from here…

Think it’s time to quit referring to Dreamhost…

Neilytron i use http://www.monitor.us/website-monitoring is a free service.

March 19th, 2013 at 3:43 pmMonty Brewster Says:

*IDLE THREAT TO LEAVE DREAMHOST*

In the order of ‘bringing back up services’, I suggest starting with my Dedicated server, especially with the DreamHost 100% Uptime Guarantee (1 day credit for each hour of downtime)

Haha my biggest question is who is hosting this website!?!?!?!

Let’s find out who Dreamhost is using and switch!…..

Thank you Derpderp! I will use them as of today! I appreciate your advice.

Neil

March 19th, 2013 at 3:45 pmMonty Brewster Says:

Its just not Super is it Mario?

Kim Jung Un must have launched a missle attack on the Irvine datacenter.

@Neilytron there are some quite handy apps if you have an Smartphone some of which are free that will run in the background of your phone and bleep when your site is offline. Have a look on the Android Market or iStore :)

Hope this helps,

Best regards,
Rob

Still not possible access my website (3:44 PM)

Has it something to do with the disk checking before the servers come up online?

I have two DH accounts, one with a single website and one with dozens of websites. Of course it’s the latter account which is entirely down right now. Great!

GAH! C’mon Dreamhost!

Is anyone back up and running now?

It’s really the thought that counts-who gives a s*** if my site is up right after people come home from work? (weeps silently)

Ouch!
My train of thought crashed.

Thanks Rob! Thanks ADMR! I will certainly look into theses!

Jak MeHoff Says:
Haha my biggest question is who is hosting this website!?!?!?!


Maybe Godaddy
:)

ok, it’s bad that all servers are down … but it was good to find a notice in the Control Panel and it is good that they update us with information. What more can they do?

Is Dreamhost expensive? No. So what are you guys crying?

Dreamhost is one of the fairest company I know, refunding you if you cancel, calculating your costs day by day. You can add and cancel items at any time. Show me such a server somewhere else.

This does not excuse the downtime – and the downtime IS unprofessional.

But I think some of you could still stay a little more calm.

Buy solar panels!!! We can use that cuz have Sun here.

March 19th, 2013 at 3:52 pmKeith Baldwin Says:

All our sites are still down here.

What FUCKKK Dudes!.. what happened with this chicken shit man?!.. i have to work in my website.. and this bull shit dont work on!!.. Please.. could you fix this problem!!

Update March 19th 3:27pm: After the power came back up ???

Our sites are STILL down! What in the world is going on here? How can a NOC go down like this?

@neilytron – No problem :)

I LOVE paying for unreliable service.

@DH: Guys, ever heard of geographic redundancy? It’s what you can do to stop this from ever repeating… Let me explain:
– You have two datacenters with constant replication between them and load-balancers to direct half the traffic in one center and half into the other. A good replication system would provide a constant sync between datacenters so that all data is replicated in real time in both locations.
– If one of the sites goes down, no problem! The other location would take all requests until the problem is solved!
– Voila! True 100% uptime! ;-)

Loosing $ every minute my site is down…specially during prom season…please get this going fast DH.

For every 30 minutes our site’s are down, DH should issue every one of us two pills of vicodin.

Folks,

I am Bobbie Lawson, CEO of Chickens N’ Skunks. Our service sells chickens and skunks and chicken and skunk accessories. We are based out of rural Alabama. Now my cousin and VP suggested we use the intranet as a way to market our goods. I don’t know much about this fancy intranet box stuff but it obvious that people live on the intranet and love our website. The website is currently down. This is not good for Chickens N’ Skunks. We have chickens running everywhere because we cannot sell them and they spookin the skunks and in return the skunks are spraying the chickens. Now this is causing the chickens to go mad of some sorts and start flying in to the air. This is causing my employees to get skunk juice all over their clothes and bodies. I have no more tomato juice left and the local grocery store just closed down due to the next county over constructing a walmart. Now I am having to drive my pickup truck to the next county to shop at walmart and that place is full of hicks. So my intranet box is down, the chickens are spookin the skunks, the skunks are spraying all over the place and now I have to waste gas to go to walmart to get tomato juice so my employees don’t stink. Meanwhile, the dog got loose and is no where to be found. Please resolve this issue soon so Chickens N’ Skunks can get back to normal.

Thank yall,
Bobbie

they can’t do anything about a power outage? If they had disaster recovery servers elsewhere they could. Maybe they do and the outage wasn’t long enough to switch, but isn’t everything supposed to be “in the cloud” these days?

I only have a little blog that get almost no traffic, but I think you all have the right to be angry.

This has happened 2x’s in 2 months. Come on guys…

“I was wondering if you can send out warnings to my email address when the server is down, and when the server eventually gets back on, like the one we are experiencing currently. The reason being is that I am using Google Adwords and when the site is down, I will get charged a lot of money for an obsolete and blank website. Once you email me that the server is down, I will be able to pause my Google Adwords account and then initiate it once the sever is ready. The amount of money I’m talking about is approximately $20-$50 per click.
Thank you for your time,
Neil”

through their mail servers? just setup a cron yourself elsewhere. you can’t rely on their mail service.

Waiting for the sun… near an hour down, but my dedicated server in the control panel says it’s up XD

My dedicated server is a liar!

Thanks Dreamhost for sucky hosting! You’re awesome!

Everyone using me is ONLINE :D

@Shaddy – My thought’s exactly. This is the first time I have had a real issue but still doesn’t effect the value for money that Dreamhost hosting offers.

March 19th, 2013 at 3:56 pmMonty Brewster Says:

@Bobbie

epic lol

We should receive an email if our site goes down as a warning. We shouldn’t have to check this blog.

Not my fault! No power!

We are still down too.

“you can check whether you site has been affected” its pretty bloody obvious its been affected because its still bloody down you moronic **** 1 hour down…1 hour of lost sales in my busiest period…..thanks…i’ll be changing host.

FUCKKKKK.. my heart is pupping like a old car.. and i think that my heart will stop in few minutes if my website dont come back nowwww!!.. Fucckkkk!.. avant chien leurs tapets!!..

Umm, okay, anytime now guys. Three VPS’s and no websites… this is getting kind of annoying.

My site is still down. Doesn’t matter. I’m alone and incel anyway. I’ll just sit and cry while male models get all the girls and servers.

I would probably not set up anything on their servers which really is connected to making money :p

But even though … the downtime overall will be maybe 2 hours, I guess. People cant tell me they die from losing 2 hours of sales. But people who shout and cry like they do here don’t look professional to me anyway.

I had a downtime on my main server with another webhoster some weeks ago. That took 6 hours to fix. And that was also something about power.

per·chance
/pərˈCHans/
Adverb
By some chance; perhaps.
Synonyms
perhaps – maybe – possibly – accidentally – by chance
——————
Heh, just had to get that one out of my system! Never get enough chance to use some of my favorite words. Hey, the server just crashed, ba-DOING!

Dreamhost status is hosted by Linode, at http://www.linode.com/
“Deploy and Manage Linux Virtual Servers in the Linode Cloud.
Get a server running in seconds with your choice of Linux distro, resources, and node location.”

I really should move my services too. I’ve been here since July 2000, and I have so many referal credits saved up its ridiculous. (Note: I no longer refer folks here. Sad.) I have an ecommerce site, and it really is a pain in the butt to move a site that has an SSL component. I’ll probably move to inMotion.com, they have the best uptime, and I’ve been testing them for speed and reliability for a few months now.

External uptime checking is really important! You can set up such checks at http://www.dslreports.com/ for cheap. Hosttracker.com is also good.

I’m bored. I wish my websites worked!

google analytics says I currently have 0 visitors online. This is good or?

March 19th, 2013 at 3:59 pmMonty Brewster Says:

@FABOX – don’t panic, have you got your towel?

First time ever I got this message, on my last post: “Your comment is awaiting moderation.”

Still down – researching other hosts as we speak. Make sure you all request your refund for your down time!

It looked like this…
Your comment is awaiting moderation.

per·chance
/pərˈCHans/
Adverb
By some chance; perhaps.
Synonyms
perhaps – maybe – possibly – accidentally – by chance
——————
Heh, just had to get that one out of my system! Never get enough chance to use some of my favorite words. Hey, the server just crashed, ba-DOING!

Dreamhost status is hosted by Linode, at http://www.linode.com/
“Deploy and Manage Linux Virtual Servers in the Linode Cloud.
Get a server running in seconds with your choice of Linux distro, resources, and node location.”

I really should move my services too. I’ve been here since July 2000, and I have so many referal credits saved up its ridiculous. (Note: I no longer refer folks here. Sad.) I have an ecommerce site, and it really is a pain in the butt to move a site that has an SSL component. I’ll probably move to inMotion.com, they have the best uptime, and I’ve been testing them for speed and reliability for a few months now.

External uptime checking is really important! You can set up such checks at http://www.dslreports.com/ for cheap. Hosttracker.com is also good.

I’m bored. I wish my websites worked!

@derp check your dreamhost contract, no refunds as no guarantee is given.

Please look into inMotion for hosting.

March 19th, 2013 at 4:03 pmRobson James Says:

If you are a company that is involved with making money, your business should have the correct “back up” procedures in place for such an event and not reliant entirely on Dreamhost.

I.e. A well advertising telephone number for customers to call you.

As my site is not for making money, it doesn’t really bother me too much.

I and my clients are losing money! Have you heard about redundancy?

Mysql servers are still down.

ONE HOUR!

Don’t worry people. Like all good web host I have back up and offload servers hosted elsewhere. I trust DH, but these things happen.

If you’re crying about “losing money” and “can’t get anything done” then you should be doing some research on backup plans.

All is well here. Thanks for the updates DH.

Derpderp, neilytron & others – Check out http://pingdom.com/ for website monitoring. It’s got a really sick interface, that will check your site every minute and will text you when your site is down. I’m doing this all for free, there are paid plans however.

Not to sound like a broken wheel but… been with dreamhost since the beginning, however over the last two years the service has gone down hill. What is going on out there?

I’m glad they are on it but downtime is happening pretty often. Pretty frustrating…

@Bobbie – funny, but if your story uses skunks, you should set it in Texas, not Alabama. There are only seven skunks in Alabama.

AGAIN?????? this happened last week! What terrible customer service. Are there tech support going to tell me that this problem is my website’s fault like they always tell me?

asshole if you don’t have power, you should have bought a electricity generator, dont you think?.. PLease god.. give me patience because if you give me strength i will kill these pieces of chicken shit

OK, competitors… lay down your special “move from Dreamhost” offers here. We’re listening for good deals ATM

March 19th, 2013 at 4:06 pmRobson James Says:

I agree that Dream host should have back-up procedures in place, such as redundancy servers. However – if you run a business you cannot entirely depend on it.

Firs big outage I’ve had in ages. Annoying, but we’ll recover.

I worked in and with data centers for many years, they’re tough, but sometimes you get these crazy awful outages and it’s a pain to come back.

Well I just went outside to start the truck and as I was pulling out of the driveway, I ran over than damn ole dog. I swear that dog wasn’t too smart to begin with. I took him hunting one day, shot a goose, it fell to the earth and the dog went and fetched me a rattle snake. Damn snake bit me in the hind end, the dog got confused on which one of us to protect and bit me in the other butt cheeck. I barely made it to the town DR and luckily he was able to save my life. Anyways, my ole lady just took my keys away because she says I have been drinking way too much beer to drive. So now I got to go bury this damn ole dog while Chickens N’ Skunks is going down hill. DreamHost, I will send you a free chieck and and free skunk if you resolve this issue asap. You have no idea what kind of neat pets a chicken and a skunk can make. You got kids dreamhost? Give them chicken and a skunk and they will love you forever!

Howdy and Outy,
Bobbie

It’s been 3.5 hours since my sites have been down. That’s a pretty long time. My clients are flipping out hahaha.

I get that these problems happen but….I have 100+ clients with sites down right now. Please give us an update (and please move me to another data center).

March 19th, 2013 at 4:09 pmRobson James Says:

@Reptar – That’s good business sense, I think some people ought to take note of offloading sites to another host in the event of a power outage – instead of relying on Dreamhost.

I can’t believe all these people bitching about all the money they are losing.

If your high-volume per hour business is “budget hosted”, you are an idiot.

Johan,

You need to get out more, son. We have a skunk and chicken farm here. We import skunk babies from other locations and breed them here. We breed brothers with sisters, with parents and so on. Most of their eyes are crossed but it makes’em a lot cuter. And hell, all my kids have cross eyes and they are cute especially when they run into stuff in the house. But I can tell you son, we have 100s and 100s of skunks and chickens on the farm here at the Chickens N’ Skunks HQ in rural southern Alabama. So if you are looking for some, please check out our site when dreamhost makes the intranet box work.

Yes’er!
Bobbie

March 19th, 2013 at 4:11 pmPissed Off Consumer Says:

This is fucking unacceptable. Has anyone heard of UPSs and generators? What the fuck is going on? Unless the fucking UPSs and generators have burned to the ground there is no reason for this. Perhaps you guys need a few more windmills to run your green system systems. Other than these shitty fucking disruptions your service is the best around when it comes to flexibility… you need to work on your up-time SLAs two times in a week.

How does one go about setting up a backup hosting service?

March 19th, 2013 at 4:12 pmMan of the world Says:

Down time is down time, if you are complaining then you did not check the reviews properly before you were bedazzled by the 100% up-time comment.

That said, what is puzzling me is…… I am currently a sysadmin in a 4th world country (never mind 3rd) currently occupied by multinational forces for the reasons laid down in history, that none of us will tolerate.

and I HAVE A FACILITY UPS FOR WHEN THE RING GOES DOWN ! do they not have facility UPS’ in CA then?

Well hopefully this gets fixed soon. Was planning to do updates and such to my website, but hope this isnt an all day thing or runs into tomorrow.

Websites been down for a pretty long time now. Not good for web ratings. Our very small customer base could go elsewhere!.

I find it pretty interesting that while our company’s website is down, Dream Host’s site isn’t…

March 19th, 2013 at 4:13 pmThe Terminator Says:

Your websites will not come back up ever because the Irvine data center was destroyed during my pursuit for Sarah Connor. It’s just rubble now. Get over it. The machines are taking over today and destroying your hosting provider is the first step in our masterplan. Hasta la vista, baby.

Regards,

The Terminator

March 19th, 2013 at 4:13 pmRobson James Says:

@Pissed Off Consumer – Chill dude!

Just a curiosity – is there an East Coast data center? Can I have my services moved out of Irvine?

March 19th, 2013 at 4:14 pmMonty Brewster Says:

Bum sex…. there I said it, you were all thinking it!

I wonder if we will get some sort of complimentary gift, maybe some adsense vouches to help get us back in the race.

TY all for the few minutes of entertainment.

I think my little blog will survive this outage just fine.

how long is this going to take ehhh…

March 19th, 2013 at 4:15 pmRobson James Says:

@Jason what you could do is backup daily your whole website MySql, web files etc and put them on another web host. Then, if you get a power outage, all you have to do is quickly change the nameservers, flush your DNS and away you go :).

Alternatively, you could make another similar domain – if you don’t want to mess about with name servers.

^Rob

#4:08 update: I don’t know about “slowly but surely”; I see “slowly but slowly”, or maybe even “slowly but not at all”.

Running a hosting company can’t be easy! I still love you guys!

March 19th, 2013 at 4:17 pmMonty Brewster Says:

@Johan hahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah……………………… etc

@dean-yes we will all get a commemorative email that says “problem happened, problem fixed, tough shit for you! Love, Dreamhost”

@Rob – NS changes can take many hours to propogate.

Hey hey everyone is being so hostile. I’m just looking at this from a business standpoint. If you pay someone to do something, they should do it. If I pay someone for new tires and they give me used, I could go somewhere else, or I could say, “I have tires so that’s good enough,” but no. I will go back to that place and not only demand my money back, but insist on free new tires for their poor service. Then I would never go back. It’s just business. Don’t charge less and say you will provide more if you can’t.

March 19th, 2013 at 4:18 pmRobson James Says:

@Jason or do what I do and use Cloud Flare which will Cache the main web pages :)

Best regards,
Rob

What is the benefit of having a dedicated server in a datacenter of poor quality?

Lets all just wait till everything is up.
Speculating and complaining isn’t gonna hurry things up for just one bit.
I’m pretty sure they will come up with a report of what happened.

March 19th, 2013 at 4:19 pmfede argentina Says:

I need to know when it’s solved. Please, we’re about an hour ago with connection problem. Is there any chance of moving the site to another server? Thank you. -

March 19th, 2013 at 4:19 pmMonty Brewster Says:

@Robson James… forgot about Cloud Flare, good call, works ok?

March 19th, 2013 at 4:19 pmRobson James Says:

@Johan – forgot about that but Cloud Flare should work on the main website pages.

You would think there would be some sort of failover especially given it’s a datacenter.

March 19th, 2013 at 4:20 pmRobson James Says:

@Monty Brewster – Yeah it works fine, I can’t access my website PHP control panel though or run any scripts but the main info I want the public to see works fine :) and the best thing is that Cloudflare is free :)

Well my sites are completely down. How long is this going to be for?

DH team, do you know when are sites going to be back up?, what is the estimated time?
I really need at least one of my websites up and running asap

March 19th, 2013 at 4:21 pmGoodbye Dreamhost Says:

Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost

You guys say “it’s cheap, so don’t complain”, but if my Hyundai stopps working, it is as serious an issue as when your BMW stops.

March 19th, 2013 at 4:22 pmRobson James Says:

@Johan – That’s a great comparison – made my day! hahahaha

Come ON !!! …. 1 hour 20 minutes !!

DREAMHOST

March 19th, 2013 at 4:24 pmMan of the world Says:

Mmmmmmmmmmmmm

Dedicated Server Dash Board – Status = Server Up (Green)

Ping website -t = Request timed out.

who’s telling lies, Microcrap or Deadhost ? answers on a postcard folks.

March 19th, 2013 at 4:24 pmMonty Brewster Says:

LOUD NOISES!!!!!

March 19th, 2013 at 4:24 pmThe Terminator Says:

I’ll be back.

Your websites won’t.

Yay. Biggest day of the week and best hours of the day. Way to go, dreamhost.

I’m very disappointed, to be honest. Unacceptable.

@Steve I was in the middle of designing when mine went down 3hours and 50 minutes ago. Yours was probably down longer than an hour 20.

The power has been up for > 1 hour (according to update timestamps, above), but still no servers. Shouldn’t take that long to sequentially boot the whole data center.

I have 2 servers (shared) that are down. Sargas and Sussex. If you have dedicated, it wouldn’t be down on those right? Maybe I don’t get it.

DreamHost, you just sent me an email that reads “We were unable to verify any server-wide problem related to the outage you’re experiencing. If you submitted a support ticket, we will be with you soon to help resolve this issue.” and then you closed my outage report as “Resolved”

SERIOUSLY?! The site is still down. How can your support people both not see that my site is completely down but also not know that you’re experiencing a major outage?

Unacceptable support response, DreamHost.

There is a new incredible technology called power backup generator that can generate power with some oil…
You can also have a high availability data center in the east side of US or somewhere with power….

@Michael – Obviously, the support team uses monitors in the data center, and they are not receiving reports from those monitors saying that anything is wrong. Or right. Or at all.

A Johnny and Susie go to the same school. One night Susie saw her mum in the shower and asked “what’s that??” She replied, “that’s the garage and you mustn’t let any pink Ferraris in.”

In Johnny’s house he saw his dad in the shower an asked “daddy, what’s that??” He replied with ” that’s a pink Ferrari and you must get it into as many garages as possible. The next day Johnny and Susie go to school and when Susie’s mum goes to pick her up, she sees that Susie has blood all over her hands, she asked “what happened!?!?”

She replied, “a pink Ferrari tried to get into my garage so I pulled out its back wheels…”

Spending money on your services and losing money due to your downtime has gotten beyond ridiculous!

March 19th, 2013 at 4:29 pmRobson James Says:

@Michael K. they can hardly do anything else….. otherwise your ticket will be clogging up their support system….they are hardly going to tell you your sites back up when it gets back online….

Dedicated Servers are also affected here…

Please, my boss kill me about that….

yep, all THREE of my dedicated are down.

This is my first month at dreamhost…. This is not worth 250 a month, im out a here, any better options ??

Well, this is proverbial straw…I hope to have all my sites moved in a week..a lot of extra work..but it’s gotta be done. fedup

My site has been down for the better part of the last 2 days. Im new to Dreamhost and not shopping for another hosting service.

I just started using the hosting here last night after transferring from Godaddy. I heard such great things…

March 19th, 2013 at 4:31 pmMatt Van Andel Says:

I have CloudFlare set up for all my key domains… why isn’t CloudFlare’s “Always Online” feature kicking in? Isn’t that the point of CloudFlare… to ensure critical parts of the website work even during an outage?

@Kate – yeah, I said the same thing during my first DH datacenter-wide outage. But now, they are getting old, and I’ll be looking.

Well this is the second time in the past month and a half, since transferring from GoDaddy, that I’ve had an HOURS long outage. FML.

March 19th, 2013 at 4:33 pmRobson James Says:

@Matt Van Andel I have Cloudflare and it is kicking in fine…

Our MySQL server went out yesterday morning, this morning and now the whole site is down. Goodbye Dreamhost.

This kind of work you have created for our clients in case of failure are living in very big trouble. All of the sites will not open. How is this a waste of energy? Commercially these results began to be afraid of. Please download these failures to a minimum. Take it easy success ..

This kind of things are unacceptable but… I still like your services

@Matt Van Andel – CF can only cache static pages. If your content is dynamic, you’re hosed.

I love the obvious posts they keep putting up like, its not up yet. We know LOL. Tell us what your doing to fix it

March 19th, 2013 at 4:36 pmMumbleBeeInc Says:

Down in Illinois. Glad this doesn’t happen to often but not good :( please hurry :)

I only actually work when your site is up. I suck :(

March 19th, 2013 at 4:36 pmEl papa John Says:

Mal de muchos . . . consuelo de pendejos.

Dale pinche dreamhost que mi abuelita no puede facturar!!!!

Come on guys get it all up and going again already it been over an hour and a half !

Correction. I do work when your site is down.

All my sites and clients sites are down – Needless to say I am getting incoming calls from furious clients!

EVERY TIME I NEED UP TIME
THERE IS DOWN TIME

Not 100% up-time -.-

My site is up and running again

I just hope its up soon up so i can move my shit elsewhere

A energy problem is unacceptable in a 100% uptime Datacenter!

March 19th, 2013 at 4:38 pmNightmareHost Says:

I’ve had nothing but trouble the month I have been on DreamHost. My other sites are on hosted on Webfaction. They NEVER have downtime and always respond to support questions within 24 hours(usually 2 hours).

Dream host always seems to be down and took 4 days to respond to my last support ticket.

@Wow – So you’re saying you got it moved to another hosting provider already?

Well I buried the dog and as I was walking back to the house to get another beer, I heard a whimper. The damn thing didn’t die! I had to undig that sob up and he just walked around like nothing happened. I swear he has part possum in’em cause he looked dead to me after my 4×4 went over his body…

My ole lady just got back with more tomato juice and our employees have been taken care of. However, I still have chickens running all over the place, spookin the skunks and you know the rest.

My ole lady bought a damn napkin holder from walmart and I don’t understand why?! I mean we did just fine laying the napkins down on the table, why does she need a holder for it? I don’t chickens and skunks and deal with broken intranet boxes so the ole lady can buy stuff that we don’t need!!!

Any word on the intranet box? My site is still showing me some error message that I didn’t make. Also can you fix my yahoo intranet mail? I put in a password and these little ****’s appear. I know I didn’t hit that key but yet, the show up. Could it be my letter and number machine or is it yahoo?

Kind regards,
Bobbie

No downtime here. You back online yet buddy?

FUCKKKKKKKKKKKKKKKK

March 19th, 2013 at 4:40 pmRobson James Says:

You are all so quick to respond when something negative happens…

Chill. It’ll be back up.

@johan U got it

Amazon cloud services are looking better and better each time this happens to us. Especially when we are in a build mode. Oh, well time to open the bar for the guys. Shots anyone? :)hic, burp!

this is unacceptable.

March 19th, 2013 at 4:41 pmMumbleBeeInc Says:

Dollar discounts for every hour off line :)right? LOL

Hey Guys – My cousin just started hosting in his garage. He only has dial-up through AOL but it’s 100% uptime guaranteed. He even has a surge protector and seems to know a lot about computers and stuff. He works at Best Buy part-time because he’s so awesome. He always about redundancy and that is why he saves everything on his computer 2 times. Speaking about up-time, he is always up, the kid is a night owl.

What do you think about migrating your sites to DialupHosts ?

This all happened right when my web designer went to install a new theme! So haven’t had a website in 3 hours and cringing!!!

i would like to know the reasons of this power disruption…. this is something we can not accept in a 100% uptime datacenter..

March 19th, 2013 at 4:43 pmINTERWEBPRENEUR Says:

@EROCX send me promocode!!!!!!!!!

EROCX,

Will your cousin take livestalk for providing intranet box services?

Bobbie

If the dreamhost datacenter approached a girl who was an 8/10, it would get rejected, because the datacenter is only a 5/10. The datacenter isn’t looks matched for an 8/10 girl. If the datacenter could increase it’s L by 1 point by getting a generator backup, but that would still only make it a 6/10.

If the datacenter got plastic surgery to correct it’s negative canthal tilt, it could raise it’s L by another point, but it would still be a point shy. Maybe the datacenter could contact Vertu, and ask him about his college friends who pull 8/10’s, because Vertu knows every intricate detail of his friends’ dating lives.

Not every datacenter can be an 8/10. Some datacenters are just born with bad genes, and are stuck leading an incel life. Meanwhile, datacenters who are male models get to date 8/10 girls every week with no problem.

Oh……..the life of an incel datacenter.

DNS changed… for good…

Dedicated server still down!

The presentation I was to start an hour ago is officially ruined. I managed to stall for an hour, but the client’s have just ran out of patience.

I know this stuff happens… but come on… one hour and half :(

@EROCX – does he have a UPS?

LMAO I had just finished some image edits on our site and went to view them when the site went out. Having to upload a new version of the webpage with each edit, I thought I inadvertantly screwed something up. I was both releived and dismayed to find out the problem is with the server and not my edits…

No power or data center redundancy? Seriously? Need to better investigate hosting providers about this shit in the future. DreamHost has been constantly going down for us for the past year.

Whether they read these comments or not, we’re leaving DH and I will never recommend it to anyone. I already have a new provider and am migrating my site this weekend.

March 19th, 2013 at 4:47 pmJames Printer Says:

Naroga: If it is an important presentation, you should have a contingency plan e.g. local web server on your laptop.

Just made a pitch for show booking at my Hotel, told the tour manager to go see my website…. looked like a fool! I had enough, dreamhost no more!

I understand that it is not your fault, but please hurry up too much time

We need a Cialis 2000 Server. Extreme Up time guaranteed.

1H 45 MIN of down time on my dedicated server… Get a GRIP dreamhost and FIX this !!

March 19th, 2013 at 4:50 pmMatt Van Andel Says:

@Johan Static pages is all I ask for… but it’s not kicking in AT ALL.

March 19th, 2013 at 4:50 pmRobson James Says:

@James Printer exactly my thought, A4 paper isn’t a thing of the past or even a flipboard!

ehh, i guess i will enable that bullshit cloudflare, but still it shows “page unreachable” if site is unreachable so i dont get it. huh. waiting and waiting and money losing and visitors turning away.
fantastic.

2 hours down – seems like it’s the end of the world!

this is completely rubbish I have been having I lot of problems with dreamhost first it was a intermittent database problem for 2 days now the power is out. I cant take this service any more. =/ only been using it for a few months.

March 19th, 2013 at 4:51 pmRobson James Says:

@Matt Van Andel Should work if you get a steady flow of visitors to your site? I would check you CF settings.

Hate to be so optimistic, but I hope all hosters leave and I have the server to myself… like we’d be virtual dedicated sort of…

I can’t- my brain can’t handle this- I sent in a job application last night linking my website as my portfolio and if they bother looking at it today they are going to think I’m a joke. I can’t believe this I really should have thrown in a pdf version but with art its just not the same. Please guys keep trying, please, or do something that can bring them up even if slow?

And again, I call your attention to the following: “…3:07pm: We now have power back in the data center.” I submit that you are not qualified to wield this power. USE THE FORCE, LUKE!!

I also have received many DOWN reports from Pingdom.. if the power is back on, why have we been down for over an hour?
http://stats.pingdom.com/kzi9qrs1pynr/675475

Matt

Snarky comment.

Yep I’m down :( I have clients needing access to my site.

I don’t understand why it is taking so long. Don’t they automatically back stuff up and can’t they redirect to the back up data? How could a hosting company this size not have that?

I installed new module on my drupal site. I think it is very CPU intensive and requested too much CPU load that requested too much electricity to compute and blew the fuses out and took the whole datacenter down.
sorry folks.
:D
seriously should investigate the uptime policy and start some requests with legal backup. this is hilarious. ads on my pages is all my income :/

March 19th, 2013 at 4:55 pmMatt Van Andel Says:

Got a response from CloudFlare support regarding their “Always Online” feature…

“Always Online (AO) will only trigger when your webserver responds to requests with a 502, or 504 HTTP error code. This means that if your webserver is not responding to requests at all (as in your case) due to a network or power outage for example, then AO will not trigger.”

In other words, CloudFlare’s Always Online feature is utterly useless for 100% of Dreamhosts outages. I need to find a new CDN.

March 19th, 2013 at 4:56 pmRobson James Says:

@June That is true!

March 19th, 2013 at 4:56 pmRobson James Says:

@Matt Van Andel I don’t understand though, mine’s working fine….

Has anyone ever gotten ANY refund for Dreamhost downtime?

I have clients needing access to my site. What’s happen??

bullshit! anybody have rec.s on an alternative?

@Matt Van Andel
A CDN won’t help you, you’d need a redundant failover web server.

Been happy with DreamHost so far, but I haven’t had much traffic anyway so downtime hasn’t really been an issue. However, just recently opened an ecommerce site, and would like to know what other good alternatives there are with better reliability? Thinking of migrating if things become more busy and I’m ready to pay a bit more.

any expepcted time to bring servers up?

still down. still down. still down.
it’s getting me down.

@ Matt

Where can I get me one of those redundant failover web servers. Wal-mart? (kidding, but seriously, tell me more?)

Seriously… an expected time would be great. I’ve spend hundreds on adwords in the last few hours for nothing. If it’s going to be 1 more hour I won’t pause but shit… if it’s going to be 5 I will. Some transparency and management of our expectations would be nice…

FUCKKKK JESUS
PLEASEEE.. CAN YOU FIX THIS PROBLEM NOWWW!!

FTP is down, website is up.

This is a bit disappointing, pretty extended outage, I think I will be moving out hosted small business apps we all use elsewhere…

This is the first time for over two years one of my sites has been down for so long. I’ve only been with dh for about 1 month.

Dean Jones – get used to it.

March 19th, 2013 at 5:02 pmMatt Van Andel Says:

@Jason

Dreamhost has a failover article that you should read… http://wiki.dreamhost.com/Failover

Worst time for downtime. We just announced a contest to win a copy of The Hobbit. :(

So… remember that one time when DreamHost said the servers were back online at 3:07 PM, and we waited until 7:58 PM EST, and our websites were still not online?

Pepperidge Farms remembers.

Glad to know I can’t depend on Dreamhost to do business…
Hundreds of people losing more money each minute…

nice comments guys lolz

March 19th, 2013 at 5:05 pmFernando Parente Says:

I need an estimate to pass to my clients!!!!!!!

Why is this impacting VA! Guess I won’t be advertising 100% uptime… anytime soon.

Please get this taken care of.

@Fernando Parente- 2, maybe 3

Thats it, Im moving all my websites to BlueHost and closing my account!!!!!!

They posted the outage notification @3:01. Posted power restored @3:07. They ain’t saying much now.

Doing some research it seems like California area 2 hours from Dreamhost in Irvine had planned outage for today!?! I hope you’re not saying this was a planned outagae that you failed to compensate for!!!? http://www.outagecentral.com/statedetails?state=CA

Has someone called to their support to get an estimate and could post it here?

Seriously guys, this is now 2h 15m into this outage. What lack of investment and competence leads to this sort of a delay? The brand continues to suffer… http://techcrunch.com/2012/01/29/dreamhosts-unhappy-january-continues-first-a-database-breach-now-an-outage/

@Alan

Wow that is dumb. I guess I’m going to get some free days of service :) lololol
http://www.outagecentral.com/statedetails?state=CA

It’s funny, had power outage at work today so came home early to find power outage at DH. Blaming solar winds? :)

So sad.

This outage is unacceptable. We recently migrated to dreamhost and I’m afraid I’m going to have to migrate *again*.

Well, I left Verio last Sunday to come here, because they couldn’t make WordPress run in a vps. Fun week!

From Monday, March 18 through Wednesday, March 20, Southern California Edison will be performing maintenance on the electrical system in the area between the hours of 9 a.m and 5 p.m. In order to perform this work safely, SCE must temporarily interrupt electric service and turn off power, according to the City of Agoura Hills website.

http://www.outagecentral.com/showurl?url=http://news.google.com/news/url?sa=t&fd=R&usg=AFQjCNGgN_6z-aJ3iEKnfD_DTLerp5qK5g&url=http://agourahills.patch.com/articles/planned-power-outages-for-next-week

I live in Florida and my site is down because of a power outage in CA?

TWO HOURS and counting. Yay.

Hey, anyone know of any hosts that have live backups in a different datacenter. That’s what the large company I work for does and I want to do it to my own sites too.

Pingas, we get it already!

Agoura Hills is not Irvine.

March 19th, 2013 at 5:15 pmMatt Van Andel Says:

My web servers (yes, plural) are back online… but my MySQL server is offline.

Ergo, all my sites are still offline. *sigh* Dreamhost, I really, really, REALLY wished I didn’t have to ever deal with this again.

Since you now have multiple data centers, if you added an automated data center cloning/failover service, I would gladly pay for it.

HERE IT SEEMS SOME LIES FROM DREAMHOST.

A LINUX SERVER DOESN’T TAKE 2 HOURS TO COME BACK ONLINE FROM A POWER OUTAGE.

Well the site is down and at least this time its not my fault!

March 19th, 2013 at 5:16 pmAnderson Pereira Says:

well, more than a your! thanks a lot for the service!

wtf????????:que pasa, tengo a mis clientes en llamas

Is it up yet? I can get it up for like four hours.

día perdido! he perdido tanto dinero hoy, que mis hijos estan pasando hambre! hoy no me pagarón un solo centavo.

———

day lost! I lost so much money now that my children are going hungry! Today I paid a penny.

It’s been 4 hours and 45 minutes since my sites have been down. sigh

this seems to be a hackers attack..

March 19th, 2013 at 5:20 pmdisappointed Says:

power redundancy?

diesel generator?

with a data center this size, you’d think that would be the norm.

I know smaller outfits with these necessary “features”

Who fucking cares about FUCKING DREAMOBJECTS FUCKING FIX OUR FUCKING SITES, FUCKING I SAY.

PERDIENDO DINEEEEEERO!!!!!!!!

Please make some effort to report 1.the scope of the outage, which appears to be national, 2. known possible reasons, and 3. best effort a resumed operation.

guess i am in the 10% of sites still down? wow, i am losing so much money! so much for paying $130/month for my VPS!

So, a major news story directly related to my blog breaks, and I have no blog, not just for a brief period but for HOURS. How much compensation will Dreamhost be paying?

PINGAS

This level of outage for multiple sites (I’m guessing hundreds at this point) is utterly inexcusable. DreamHost, you are screwing with people’s incomes. There had better be some sort of major, very personal apology coming at the very least…

The more they talk about DreamObjects, the more I hate it. I don’t even know WTF that is.

Why do i care about dreamobjects, my websites are down?!

PINGU

good news.Nothing to ease.

OH and by the way, one of those sites is the information site for a troop support center, so I know at least on my end you’re messing with families and returning service members… FIX THIS, PLEASE.

I got a “dreamobject” for you right here.

March 19th, 2013 at 5:25 pmMumbleBeeInc Says:

bah hum bug!! I go info that has to get to about 35,000 peeps I am running out of time :(

At a minimum we should get refunded for down time.. I need my website up and running!!
I thought this host was 99.99% reliable..

DreamHost Status: What’s Up?
YOU MEAN:
DreamHost Status: What’s Down?

I have heard good things about LiquidWeb (www.liquidweb.com). Anyone had experience with it?

‘##::: ##:’####::’######::’########:::::’#######::’##::: ##:’########:
###:: ##:. ##::’##… ##: ##…..:::::’##…. ##: ###:: ##: ##…..::
####: ##:: ##:: ##:::..:: ##:::::::::: ##:::: ##: ####: ##: ##:::::::
## ## ##:: ##:: ##::::::: ######:::::: ##:::: ##: ## ## ##: ######:::
##. ####:: ##:: ##::::::: ##…::::::: ##:::: ##: ##. ####: ##…::::
##:. ###:: ##:: ##::: ##: ##:::::::::: ##:::: ##: ##:. ###: ##:::::::
##::. ##:’####:. ######:: ########::::. #######:: ##::. ##: ########:
..::::..::….:::……:::……..::::::…….:::..::::..::……..::

I mean, nice one.

This has to be a joke.

I am really frustrated with this company.. I havent even been here a year and I can’t count the number of times my site has been down for unknown reasons.. I realize this time there is some reason that of course we do not know but where is the 100% up time guarantee?? Seriously unhappy here and my sites are still down.. wtf??

Perhaps it’s time for dreamhost to consider moving their data centers out of areas imperiled with brown outs (yes, I’m talking about you, SoCal)…

My site is still down and the chat support is disabled from my “control” panel

I’m sure they are starting with shared hosting because most people will use this plan. For those of us with VPS plans, I guess we are the bottom of the barrel and will experience the longest outage. I have to agree though, why in the world would they get DreamObjects up first?

While I understand that downages are problematic. I pay a lot and really dont care about DreamObjects. I need restoration now.

Come the fuck on already… is this going to be fixed soon, or should I stop dumping hundreds of dollars on adwords for traffic that will never see shit? It would SURE BE FUCKING NICE TO KNOW. Oh wait, apparently shit got fixed 2.5 hours ago… NOT.

DH is using a XT to host my website. 2 hours to restart ?

Come on we’ve been down for hours and hours now!!!!!!!

Seriously who the hell cares about DreamObjects. Fix the damn database servers first.

still down, but I know you guys are getting to it as quickly as you can . . . ignore those on here complaining about the refunds and other stuff . . . your hosting is the best/easiest I work with

Thank you for what you all do day-to-day . . . technology goes down . . . that happens, nothing you can do to stop it

Are they stupid? Who care about dreamobjects when the websites are down?

It is not acceptable to have a service interruption of more than 2hrs for a power disruption. Especially if the power disruption was planned.

Question: what are the most reliable host services on the market?

What about the 100% up time guarantee??

Seriously, I’m losing money with my sites down.

Shauna, Oh crap… I’ve only been here 3 days!

Fuck this. Paused that shit.

This is the last straw. I’m actively looking for a new host now. You can kiss my $1,000 a month payment goodbye you fucking assholes.

where is yours power generator??
My Backups is DOWN….because DreamObjects is DOWN!!

Do NOT go to Verio. Poor infa and support.

Private servers should be the first thing to come back on!!!!!!!

I’m actually ok with it being down for a bit, but I must say it is damn ironic that the first time in MONTHS that I have decided I am going to post a blog entry, this is what I find…the universe must be telling me something, either about my writing or my tendency to procrastinate by finding things, like writing a blog entry, to distract me from my other less palatable work. :-/

Well my dedicated machine is Up and Running, even restarted it. Pity it doesn’t seem to be connected to the outside world.

And like others, who cares about DreamObjects I care about my Servers and DB Servers. If I can host a static page I can at least do something to cover up this mess.

P
I
N
G
A
S

I’m an IT guy and I’m even getting pissed off. I’ve got our CEO pushing me to move our site to another host TONIGHT.

come on!!!
Estimated time please

I love the way you lie DreamHost, I love the way you lie

This can’t be a power failure. Someone got into their system.

If I had lots of traffic on my site, I would be very unhappy now. So, they advertise “The DreamHost 100% Uptime Guarantee”, now that it’s obviously not 100% what do we get since you break that guarantee?

Our website is still down here in Irvine, CA.

I am online yet

My servers are still down.

not yet

el día de pago les voy a decir que se me fue la energia y no pude pagarles

Fuck this. Paused that shit.
This is the last straw. I’m actively looking for a new host now. You can kiss my $1,000 a month payment goodbye you fucking assholes.

We should host with Cirtex Hosting – that’s where dreamhoststatus.com is.

I give up. :( I am going to move my sites. There have just been too many problems here. Makes me sad. Any feedback on Hostgator, guys?

We’re still down. Any updates? Should I fire up XAMPP on my laptop :P

So there was a planned outage yet DreamHost didn’t prepare for it really. This is just shocking. Come on.

What does it mean to have “DreamObjects are now available”?, my site and my vps is down!
You need a new concept called “NightmareObjects”

Any updates about the Dedicated Servers?

I just logged into my dreamhost account and it doesn’t even mention the power disruption anymore yet I still can not load my site not get onto my wordpress admin page… Looks like I’ll be moving to bluehost when my dreamhost plan is up. This is the first time my site has been down but this is unacceptable.

@Rihanna You better shut your mouth, or I’ll shut it for you. Loved your comment tho.

Im still down.

any idea for a reliable Drupal managed hosting ?

I can’t believe people are paying >$100/mo for this service and sound like you have big businesses going on. Luckily I only pay peanuts for shared hosting at DH. For any serious business I would never host my website on a budget provider.

Update March 19th 5:34pm: At this time, we’re still working on the problem, and are going through the many troubleshooting steps needed, however these steps take a bit of time to complete. We’re working as fast as possible to get things sorted out ASAP.

still down too.

Come on just hit the damn power button.

This is a nightmare, you are damaging us.

I heard Westhost was good

Troubleshooting?!!?, how about buying the book HOSTING FOR DUMMYS, $10.99 in all good bookshops. This is ridiculous!

I’m very disappointed also!!! Unacceptable. I have new clients to test my system and they can’t because my server is offline! How can I explain this to them???

…and now all the fans of NSFW Show on TWiT are noticing that nsfwshow.com is offline. The streaming video show starts soon. Thanks for this Dreamhost. You’re great. Just great. We needed this.

Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!
Ridiculous !!!!!

come on!!!
Estimated time please

INSTALL POWER BACKUP SYSTEMS.

Down :(

Is there an ETA on this I have 10 sites down.

Seriously, people? This is the first serious outage I can remember from Dreamhost in like the last two years, and it has lasted two and a half hours. Relax, get over it and let them fix it.

Good thing I wasn’t running a March Madness pool website. Oh wait… I am. More than 100 DH websites, and ALL down. Very bad timing.

JT this is the THIRD outage I have experienced in my 6 months or so here.

Can you please give us an estimated time we’ll be back online. We all REALLY need to know.

Could you please at least give an estimated restoring time?

THis way we can plan migration tasks.

Still have my dedicated server completely down.

Would it make any difference if I added VPS account? Would it get my sites up?

Diamond Club represent!
<>

Hire people who know that you should always have 2-3 redundant connections with different service providers with fail over. You’ve been in business how long and still don’t have it set up?

Man I know this is a once in a lifetime issue and I want to be patient but I’ve lost a client. That’s $4000 in annual billing. It’s been a great run but this is NOT 100% up time. We have other accounts with Dream host and they are up now ~ as a former network engineer I can appreciate the situation but this is an absolute no no.

CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!

“This is the first serious outage I can remember from Dreamhost in like the last two months, and it has lasted two and a half hours”.. Corrected that for you JT

I wish i would have my money back, I feel so sad with you guys =/

FIX THE PROBLEM ALREADY!!!

CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!
CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!

CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!CRAAAAP!!!

Both our sites are down. at 8:45 PM East, 5:45pm West

Bit ruff when downtime lasts several hours for us. Most of our traffic comes after 5:30 PM here

……………………………………..________
………………………………,.-‘”……………….“~.,
………………………..,.-“……………………………..”-.,
…………………….,/………………………………………..”:,
…………………,?………………………………………………,
………………./…………………………………………………..,}
……………../………………………………………………,:`^`..}
……………/……………………………………………,:”………/
…………..?…..__…………………………………..:`………../
…………./__.(…..”~-,_…………………………,:`………./
………../(_….”~,_……..”~,_………………..,:`…….._/
……….{.._$;_……”=,_…….”-,_…….,.-~-,},.~”;/….}
………..((…..*~_…….”=-._……”;,,./`…./”…………../
…,,,___.`~,……”~.,………………..`…..}…………../
…………(….`=-,,…….`……………………(……;_,,-”
…………/.`~,……`-………………………………./
………….`~.*-,……………………………….|,./…..,__
,,_……….}.>-._……………………………..|…………..`=~-,
…..`=~-,__……`,……………………………
……………….`=~-,,.,………………………….
…………………………..`:,,………………………`…………..__
……………………………….`=-,……………….,%`>–==“
…………………………………._……….._,-%…….`
……………………………..,

That’s it. I am out of this goddamn server. Poor service, high prices. Going to somewhere else. ¬¬

An ETA would really help out. So that we can explain to our clients.

What hosting company do you guys suggest?
I’ve had enough.

Hostgator, Bluehost, [add your suggestion]?

Website down!

Mi sitio no funciona!

Soy nuevo en DH. Esto es frecuente?

All my websites / web applications and also my custmers websites are down!

My shared host sites are all up but my VPS is still down. I want a credit to my account.

It’s frustrating that the panel.dreamhost.com has taken down the “power outage” issue all together — yet wezen (and I presume others) isn’t up yet.

I’m not complaining about the level of service here — I’m not expecting ‘cloud’ computing, or virtualization that fails over to the east coast, etc.

But I *DO* expect the human element not to “withdraw” and make nice like all is back to normal — when there still are MAJOR outages on a server that hosts hundreds of websites, including ours.

This is Red Leader. Report in. Over.

Red Leader, this is WTF. Maday. Maday. Do You Copy? We’re Going Down!

March 19th, 2013 at 5:58 pmEl papa John Says:

Mi buen guerrero, no es tan frecuente, o nunca había habido un downtime tan tan tan largo. Y lo bueno es que no está ancho porque sino nos revienta de plano.

After 13 years of being with Dreamhost, and moving to a private server when this happened in the summer I must say it is now time for me to migrate. So sad. But the time has finally come. Just doesn’t make business sense to stay loyal after the number of mess ups that have been happening.

THREE HOURS!

H. Guerrero si, si es frecuente lo mejor es que tengas una segunda opción…

No te voy a mentir, Dreamhost tiene una bonito panel de control y todo muy funcional y buenos precios, pero todo eso tiene un costo…

Haciendo un conteo, en un mes, 10 dias tu página va a estar caida.

Our site has been down for three hours now and doesn’t appear to be coming back up anytime soon. I really hope DH loses as much business over this as we have… With a marketing budget in excess of $20k per month, it makes no sense that our website’s status should be dependant on a bunch of defective DH servers. I think it’s time to start looking for a seemless transfer to another host.

Copy that WTF. Deploying Recovery Units. Stand By.

I can’t complain since I’m getting the non-profit discount. But still, they should have implemented some type of load balancing.

Any word on Baptisia? Still can’t get in.

Website is still down. After seeing that this could have possibly been a planned outage and dreahost (most likely) knew about it, I think I may start looking into other VPS’s.

I’m just not sure which one yet.

El papa John, tu como que no eres de aqui. jajaja como vas a decir que no es frecuente, tan solo mira el monton de publicaciones que hacen en este blog.

yo un VPS, VPS MYSQL y contrate el nuevo servicio de DreamObject y en estos momentos todo esta caido.

How could this possibly be about a power outage? That would not take three hours to work around.

I fucking hate you dreamhost.

Our site is down. If 90% is back up I guess that means we’re part of the less fortunate 10%.

I’m in LA and we have not had any power outages here.

Further, wtf clients does everyone keep replying about?

What kind of business’es are ya’ll running ? some sort of “reseller?”

… lingering issue is a routing table we’ve been unable to restore…

Router is the guilty

=]

I’m in Australia and its midday. Our site is still down and we are using the 2 weeks free option. I think I will have to look elsewhere!

Have you tried turning it off and on?

March 19th, 2013 at 6:05 pmEl papa John Says:

Bueno tanto así como un downtime del 30% no creo, si hay downtimes de 1 a 5 minutos 15 como máximo.
yo tengo varios sitios y no hay tanto problema, pero definitivamente creo que no es mala idea tener dos cuentas chafas (en dos distintas compañias y buscar una forma de hacer switch entre ambas, pero creo que esto deberá tener un coste en velocidad y los certificados SSL no es tan simple de mover de un lugar a otro.

WOW, hasn’t even been a year with DH and my site has been down 4 times now, this has been the longest it’s been down though. I moved here on a recommendation from a friend but so far has proven to not be the smartest move.

Does it make sense that only a part of a website would be down (blog pages only)? Sluggish to boot.

Glad my press release didn’t go out today!

Hi Guy’s

I’ve been up since 3am AEST this morning organising a server migration which was scheduled to be completed by 8am AEST.

it’s currently 11am AEST and my client is breathing down my neck to get the DNS updated.

I also noticed that CHAT LIVE is not available on my account – I have posted a support ticket with no response – can some one please give me an update on the status of migration at least let me know it has been canceled so that we can reschedule

Thanks for letting us know what is going on.

@John: IMPAK Corporation (sorbentsystems.com) is a manufacturer and distributor of sorbents, desiccants, mylar packaging and sealers based in Los Angeles. We have been in business for over 20 years and sell worldwide. We are not a “reseller”.

DH staff is busy looking for the power switch, don’t disturb them so we can get back online!

Dat wun dare bee a gud eye deer dat wun.

Down toooooo LONG! My VPS is down and I care – I am one of those people that makes an income from my web site and I am NOT happy that it has been down this long.

(╯°□°)╯︵ ┻━┻

This is exceedingly frustrating. We have one big night a week and of course, it’s tonight. I was not worried because it didn’t start until 9pm eastern, but yep still not up. All my crappy little sites that are on shared hosting are up, but my VPS site is still down.

Now I’m trying to do what I normally do on a 5k+ member forum on facebook…wheee

still down too.

So my cousin, Daniel Barden was killed at Sandy Hook Elem on 12-14. We launched a website and Facebook page to honor his life and memory. (WWDD) http://www.facebook.com/whatwoulddanieldo and http://www.whatwoulddanieldo.com. We started selling WWDD bracelets last night and have had over 200 orders and now nothing as a result of this. With WWDD on my mind, I am just going to trust that DH is doing their best and thank them for a great affordable service! I thought I would try to bring some light into this dark place. Cheers! :) You can do it DH… WWDD believes in you!

@Jameyson MacDonald

Ahh thanks for answering my curiosity. I was thinking more along the lines of say a web programmer/designer (which is what I do) and wondering how the rest of these people were getting such high volume of clients and pay/hr

Also, the irony of me trying to check your site out only to realize it’s down lol

3 hours down, what is dreamhost going to do to help us repair de damage done to our brands and webpages?

Interesting. Have not even been with Dreamhost for two months yet and this is at least the third time it’s been down for more than an hour.

I FIRST EXPECT MY SITE UP AND RUNNING ASAP.
THEN I EXPECT A COMPENSATION AND FULL DETAILS OF IRVINE DATACENTER SPECS REGARDING POWER REDUNDANCY.

IF I’M HOSTED IN A DATACENTER WITH NO AUX/EMERGENCY POWER SUPPLY, IT’S LIKELY IT WILL HAPPEN SOON AGAIN AND I’M NOT KEEN OF SUFFERING IT TWICE, AT LEAST FOR SOMETHING BASIC.

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

I appreciate DH letting us know about the routing table, but I am still confused why they cannot transfer traffic to back up place. Or don’t they have a back up place?

I think I have a solution. move everybody’s websites to the server that this blog is on. I have never seen any outages here.

Well enough of trying to save a buck on discount servers. I deserve this. This is all my fault. I just thought that 100% uptime
ment something else.

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

1………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

Still down for me :( Would be nice if the VPS’s were mirrored to more than one data center, so if something crazy like this happened, it would fail over to the mirror in the remote data center automatically!!!

-Jamie M.

FIX THE PROBLEM ALREADY!!!

4 hours and counting.

I don’t understand how the shared sites are up but not the VPS or Dedicated boxes.

A haiku for you Dreamhost folks:
The website is down.
People thinking it’s my fault.
Blame Dreamhost again.

I’m not hosted on a DreamHost server.

Still down…what’s going on?

I’m not losing money, but I am paying for this service. There are free services up that have better uptime than this.

@john
the shared sites are hosted in a different datacenter located in a different, distant place from IRVINEGAR that gives us that bad taste everyday.

FORA do ar todos meus sites!!!!

FORA do ar todos meus sites!!!!!

What’s a good alternative to Dreamhost? I’m honestly sick and tired of this crap.

I’m on a shared, still down

@El papa John
@Murray Greer

Gracias por las respuestas.
Tengo tres dominios alojados en DH, pero no son para mantener clientes o ganar dinero. De todos modos me sorprende que se caiga tanto en un mes!!! Estoy desde febrero y es la primera vez que me entero de que el sitio está caído.

compensation would be good.. Il have $200 an hour please.. where do I send the invoice?

March 19th, 2013 at 6:18 pmRecommendations pleas Says:

In all seriousness, can anyone recommend a good alternative host. I need to move my sites onto something reliable. This is not good.

PUTA QUE PARIU!!!!

I am losing all confidence of people in my site. I am delivering online services – I can not be down. C’mon guy – what is wrong with you????!!!

Gracias! a dios! esto no fue en la mañana de hoy!!! y espero no llegue a mañana!

My sites are on shared servers and they have all been down for 4 hours now.

Website is still down.. The one day we get a really good blog posting hosted on another design site and the photos are hosted on our site..

Bluehost.com is looking really good right now.

I am trying to determine how long my site has been down. It looks like it could have been the entire day.

buy some fucking solar panels and batteries!

They must not have a back up they can direct to which is crazy. I just don’t get it.

all my sites are still down. :(

Header on this page says “DreamHost Status What’s Up.”

. . . . Ummm….. NOT OUR SITE!!!!!!!!!

My clients are FURIOUS with me…. and it’s not my fault :(

This is unfair….

meus servidor dedicado já esta off por três horas

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

Who hosts dreamhoststatus.com? It’s always online! :)

En el mensaje de las 5:10pm: dice que el 90% de los sitios funcionan, pero de mis tres dominios ninguno tiene la suerte de estar en ese 90%.

Alguien es de los dichosos del 90?

bueno no se si los servidores compartidos se caen tanto, pero lo que es VPS o servidores virtuales, a cada rato se caen o bueno por lo menos los mios.

yo vivo de esto, tengo 34 páginas alojadas y creciendo, que facturan y todas estan caidas.

Still not working for us. Please let us know how much more we have to wait till server comes up again.

In the Midwest, and my site has been down for more than four hours. We’re an online news site, trying to build clients and advertisers. So on a big news day following some local decisions that I reported on, viewers are getting their news from my competitors, who have not been offline during the 8 months I’ve been with Dreamhost. This is at least the third time we’ve been offline and often experience very slow response to loading the site. This is not good and will start looking at options. DH was highly recommended. Where’s the backup to the bakup to the backup? Should be prepared for a power outage.

@Paul

Don’t go to Bluehost. I came to Dreamhost from them and this is better than what I got there.

March 19th, 2013 at 6:24 pmEl papa John Says:

Busca otro post en dreamhost status con esta cantidad de comentarios y/o tiempo fuera de línea dentro de este mes y me trago cada palabra. Es más te la valgo en todo el año. Así como que un 70% up y un 30% down se me hace exagerar. Francamente si debería de tener dreamhost un respaldo al menos en 2 datacenters iguales para evitar estas situaciones. Pero eso es negocio de ellos no mío.

We are still down. Going to submit a support ticket in a little bit if we don’t come back online.

WhoIsHostingThis website shows Cirtex Hosting as the hosting provider for DreamHostStatus.

March 19th, 2013 at 6:25 pmM Schillereff Says:

It makes no sense to at all how it takes this much time to bring things back up after a power outage, but I would be asking myself, how on earth did a power outage bring things down to begin with? Does the Datacenter not have backup power, if not remind me to move my hosting someplace that at least has back up power. I host outside because I don’t have a backup power system, but I think it may be more secure to bring it back in house.

tengo a todos mis clientes putiandome como es posible que esten demorando tanto en resolver el tema
exijo que la repuesta sea rapida
llevo ya harto tiempo con ustedes y siempre hay problemas

Antes me arrancaba los pelos de la cabeza, hoy día solo puede reirme y relajarme, nada gano con agarrar a puños la pc.

y lo mas tirste de todo es que estoy tan acostumbrado a la forma de trabajo de Dreamhost y en la forma como manejos mis VPS que me tienen amarrado a dreamhost.

pero recomiendo a los que estan comenzando que tengan pr favor una segunda opción.

qua qua

Most of us are losing money. We cant do anything about it at this time. Let them do their job & figure it out. I’m sure they will take care of us in some way once services are restored.

What kind of a datacenter doesn’t have a UPS power backup? cmon already…

Doesn’t any network/system administrator backup the routing table on a regular basis? this is insane! the guy should be fired!

March 19th, 2013 at 6:28 pmEl papa John Says:

Por lo pronto a pedir disculpas y a bajarme los chones con los clientes. Tampoco estoy en ese dichoso 90% de que ya funcionan, ni otras cuentas de recomendados. Ha sido un corto muy largo maldita sea.

@Murray Greer
Que pena, si trabajas y vives de eso es fastidioso!!!
Yo pensaba que los VPS eran mas fiables… O la ventaja que tienen es la rapidez, IP fija y eso, sin embargo se caen cuando hay un problema en el proveedor? En todo caso imagino que los VPS tendrán una cierta prioridad al momento de resolver un problema. Que mi sitio dure algunas horas down no me afecta en nada, pero para quienes viven de esto, es delicado.

Bueno esperemos que se resuelva pronto.

Status please!!!! site down :(

A todos los hispanos, todos los servidores del mundo tienen problemas. Es cierto que Irvine en concreto y en el último año ha tenido más problemas de los habituales (sobre todo de DDoS), y que un fallo de suministro eléctrico es algo básico en un datacenter.

Yo pido que Dreamhost aclare si tiene power backup en Irvine o si no lo tiene. En caso de tenerlo y haber fallado el sistema de conexión alternativo, que lo chequeen (shit happens) pero si definitivamente, es que no tiene backup, esto es MUY GRAVE.

Irvine es el hogar de los VPS y de los dedicados, principalmente. Esto es, de los que más pagamos.
EXIJAMOS UN BUEN SERVICIO.

Aclarar que el 90% se refiere a otra cosa, no al número de sitios que funcionan.
NINGUN SITIO FUNCIONA ACTUALMENTE.
EL 90% SE REFIERE A OTRO SERVICIO QUE EN ESTOS MOMENTOS A NADIE LE IMPORTA QUE FUNCIONE O QUE SE HAYAN PERDIDO DATOS.

March 19th, 2013 at 6:32 pmAnderson Pereira Says:

Caralho, cortaram o gato nessa porra!!! To perdendo grana caralho, como vou vender meu pó online?!?!?!?!

March 19th, 2013 at 6:33 pmAnderson Pereira Says:

Dreamhost, no mínimos 100 dólares pelos serviços de hoje! Vou atrasar o lançamento do meu site por causa de vcs??!?!? Cadê o gerador?!?!?!

Todos mis sitios caídos. HORROR !!! Una verguenza.

March 19th, 2013 at 6:34 pmMatt Van Andel Says:

@watch_iTV

Having been through this before, I expect little more than a shrug from Dreamhost.

And in the mean time, what about those of us with angry clients and customers of our own? To them, this is our fault… and in a way, it absolutely is. We’re screwed.

March 19th, 2013 at 6:34 pmone of DH user Says:

FIRE STORM …!

I’m angry, though.

@ jose miguel olmos

No sé si te leerán… Leiste lo que ponen justo encima del cuadro de comentarios?

Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

Ademas estarán bien ocupados reparando la routing table.

Una pregunta a los más expertos, este tipo de problemas podria provocar pérdida de datos en nuestros sitios? Tengo un mes con ellos y no he siquiera tomado el tiempo de hacer un backup de mis datos por ejemplo con WordPress.
Justo hoy hice un Backup desde el CPanel y recibí un fichero por email, eso me serviría para restaurar todo en caso de problema?

March 19th, 2013 at 6:35 pmEl papa John Says:

Si trololo está complicado yo tengo vps y me había funcionado mejor hasta ahora, pensaba cancelarlos debido a los costes y que pude sustituir algunos algoritmos y pasarlos a base de datos, pero de cualquier manera es complicado, yo he estado con varios y ciertamente es imposible que te garanticen un uptime del 100% pero esto ya estamos hablando de mucho tiempo. Normalmente son 2 a 15 minutos y se levanta el servicio. Como dice por ahí el buen Guerrero lo bueno es que no fue en la mañana, que es dónde tengo más tráfico.

We;’re down. Very frustrating. We just sent out an email blast. I know you are working hard on this, but please fix this quickly. The constant outages are becoming very problematic.

March 19th, 2013 at 6:35 pmDavid Dewhirst Says:

One free day of hosting for every hour down: http://idreamhost.net/uptime-guarantee

Tudo fora!!! Nossa os clientes tudo buzinando, meu celular nao para de tocar!!!! Bricadeira Dreamhost, ajeita isso aí!

From http://dreamhost.com/web-hosting/

The DreamHost 100% Uptime Guarantee
DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that!

SE FRIERON LOS ROUTERS.

:D ——>>> http://www.bluehost.com/track/clickable

BLUEHOST WINS!

I have a dedicated dreamhost that is down A LOT!
Bluehost haven’t been down once in well over a year!

:D ——>>> http://www.bluehost.com/track/clickable

March 19th, 2013 at 6:38 pmEl papa John Says:

Trololo, no creo que se pierdan datos, salvo los formularios que se estuvieren cargando al momento del apagón o archivos que se estuvieran transfiriendo, ya que normalmente trabajan sobre discos rígidos. Y si ya se que los dreamhost ni se enteran de estos comentarios (pero sus mamás seguro que sí), insisto mal de muchos consuelo de pendejos, tan siquiera aquí podemos descargar el veneno para que no se nos amargue tanto la hiel.

Update March 19th 6:30pm: As of now we have two network devices, that handle traffic internal to the data center, that are down. One main one, and one backup. Our vendor has said, that BOTH devices were fried during the power outage, and are suggesting we RMA BOTH of them. We are in the process of deploying a spare to that location and we estimate that will take 1-2 hours. We will continue to update as we get more information.

——–

More 2 hours … WTF

Six hours and counting

HOLY SHITTTT.

Another 2 hours + install time.

“We are in the process of deploying a spare to that location and we estimate that will take 1-2 hours. We will continue to update as we get more information.”

WHAT?! “in the process of deploying a spare to that location” = going out and buying one and driving it across town.

SOMEONE GO BRING SOME PIZZAS.

Visions of Clowns stumbling around in the dark….

DreamHost should look into getting some APC equipment. This is crazy!

March 19th, 2013 at 6:42 pmEl papa John Says:

Si los routers están fritos y la facturación de mi abue del día de hoy también. Creo que alguién va a salir despedido de dreamhost y exijo mis 3 días gratos de hosting. (como si eso fuera algo)

Thank you for letting us know what is going on.

I hope we will be reimbursed for this outage.

Hey! Try http://LawFirmSearchEngine.com for free legal research! Oh wait, try again when the server is up!

Can anybody recommend any reliable VPS hosting company? I want to move a few sites…

WHAT IF THEY CANT SOURCE BACKUP HARDWARE TONIGHT? WILL WE BE DOWN FOR DAYS?????

PROLONGED DOWNTIME CAN DEFINITELY IMPACT SEARCH ENGINE RANKING!!!!!!!!

NOT GOOD.

Does anyone know of a better hosting service? I am shopping around for one and I know I am not the only one who will be doing a transfer before this night is over.

Ya andan por ahi haciendo publicidad a la competencia. Lei por ahi Bluehost???? No, por Dios. Soy principiante en esto del hosting, estuve 1 dia y medio con ellos y me cerraron mi sitio. Luego me pedían copia por fax de mi tarjeta de crédito o llamar a un numero en EEUU, y yo residiendo en Francia. No, Bluehost no por favor. Hasta el servicio comercial fue mediocre cuando traté con ellos. Y no les importó perder un cliente que hoy día va por 4 dominios contratados y un plan de 3 años de hosting con DH. No es mucho pero un cliente es un cliente.

Que viva DH y que suba rápido, vamos DH arriba!

De mis 4 dominios, 3 de ellos en Irvine y 1 en Los Angeles. No problem en Los Angeles!

So Dreamhost… This is 100% Uptime? My site has been down the better part of 2 days. Not impressed and losing clients.

The DreamHost 100% Uptime Guarantee
DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that!

http://www.bluehost.com/track/clickable <<<—– sign up here for much better uptime than this dreamhost bullshit!

“…RMA the both of them.”

Typed as if I know what the hell “RMA” means.

ugh ALL my sites are down and i’m getting some angry emails from clients, please hurry

FOUR HOURS!!!

ECOMERCE DOWN ! WTF !!!!!!!

@WC Return the items.

I do NOT care if you need to RMA the fried devices. Get this BACK UP PLEASE.

March 19th, 2013 at 6:49 pmEl papa John Says:

¡Les doy la tarde libre a todos!
duerman tranquilos que otros tienen que sacar este problema.

Quien sea libre de pecado que tire la primer piedra.

Hard to believe that *both* network appliances were fried in the same surge. Not hard to believe that it could happen, but that that both primary and backup were powered through the same circuit. Foolishness, someone should lose their job over this.

BTW, inMotion.com has good service.

we are still down…Please forward me expecting time.

…and my VPS still DOWN!….(SUX!)
…and the DreamObjects still DOWN to!…(SUX! (again..))

Ataque terrorista en DH?

Terrorism in DH?

Y toda esa publicidad de la competencia?

March 19th, 2013 at 6:53 pmAutomation Client Says:

CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!CRAAAAAAAP!!!
Y………..Y………..Y………..Y………..Y

Guys, you should consider switching to another host like inmotion or mediatemple. You can have your websites switched over by the time it takes for them to get your stuff back up. You are looking at huge downtimes here.

Isn’t it ironic how http://www.dreamhost.com is up and running smoothly, they must be using amazon s3 hosting.

down1!!!

March 19th, 2013 at 6:55 pmEl papa John Says:

Como todos están poniendo sus comerciales, bueno vendo pupusas, muy buenas, recién hechas, de chicharrón, pollo y también de picadillo. Con salsa de chile y toda la cosa. Entra a http://www.ricas-pupusas.com y haz tu pedido.

Chin, ese sitio también está caído.

Seriously? Come ON, Dreamhost. You’ve screwed us before with your unreliable service and now I’ve just had a client sign up for your hosting.

You either need to completely revamp your system and the way you do business and promise all of your customers that this will NEVER happen again in writing, or you need to offer people a way to get their money back and break their contracts.

If you’re not going to offer what you claim to, at least be honest enough to let us walk away with some dignity.

coño! hasta cuando con esto! yo si quiero que me regresen mi dinero perdido en estas 5 horas!

Just signed up for rocketVPS.

Using their promo code Rocket25LIFE I’m now paying 11$/month for LIFE for 512 MB of ram AND one dedicated IP address

Only downside is you have to install everything… apache,php, mysql

Two more hours????? RMA????????

hmm let see if the billing reflects this:

From http://dreamhost.com/web-hosting/

The DreamHost 100% Uptime Guarantee
DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that!

Revenue is gone…

March 19th, 2013 at 6:57 pmanother unhappy customer Says:

Who knew a dedicated server would get you great service like this? Amazing!

I can not believe what a rediculous situation this has become. The timing could not have been worse for my clients big campaign launch today and into tomorrow. You’ve absolutely destroyed the credibility of my business and I can not wait until morning to start moving my entire production system over to Amazon. This is absolutely ridiculous and I hope every person over at Dream Host that is responsible for the massive screw up is fired immediately. Shame on you Dream Host!

Dam it… 2 more hours.. come on, this it too much

I do need my VPS hosted at the same server where dreamhoststatus.com was hosted. NOW!

Make sure to submit a ticket in order to receive your “one day for every hour of downtime” credit. Its part of the fine print.

Qué es eso de enviar un ticket para que te devuelvan en dias de hosting las horas que el proveedor ha estado caído? Es que yo entiendo un poco de inglés pero no todo…

@Nikki J – why would any of us even want to get a credit from such a horribly unreliable company? It’s not even worth it.

Would I be able to forward my domain to a temporary site. I set a redirect but it has not taken. Thank you.

All of my sites are down. No FTP no email. My clients are pissed.

I don’t normally whine, but damn, I am losing money. *crys into arm*

That is very bad, when can we expect this fixed. All my sites are down and it’s a big launch day.

Servers down for 5 hours now (and counting). Expect a discount on my VPS & DreamObject bill

March 19th, 2013 at 7:02 pmEl papa John Says:

Guerrero, es una burla, te van a reembolsar un día de hosting por cada hora de downtime, este downtime va a durar algo así como 6 horas, osea 6 días de hosting. pero si divides 119 dólares del año entre 365 días te sale el día como a .40 c de dólar que sería como 2.4 dólares de reembolso. Hasta dónde yo lo veo pues no es una super-ganancia más para alguien que tiene una aplicación formal subida y tiene que brindar una atención a sus clientes y abuelita. Pero de eso se trata la garantía del 100% up-time.

RMA = Rape his Mother Again

All I can say is I’m glad we only host our wordpress site with you guys and not our actual online store. What a mess, guys. You don’t keep backups hardware for your most critical service (hosting) on site at your primary west coast facility? Or switchable backup power and surge protection on your hardware? Is this amateur hour?

Not to complain along with everyone else. I can understand that things don’t go as planned. But I believe the staff could at least throw up a mirror of the effected sites and offer some sort of message telling everyone that things are down and being fixed. There should be backups like this in place, if the main backups don’t go as planned. Everyone that visits an effected site atm will just get the basic browser error, for all they know…your site was shut down (and no one wants to lose visitors like this.)

Pissed Off? Send your comments directly to the CEO of Dream Host and give him a piece of your mind — > @DreamHostSimon

Geezz, dreamhost, from the looks of things on here this little outage is gonna cost you some clients too.

THIS IS RIDICULOUS!!!!! LOSING MONEY HERE. UGH CHANGING TO BLUEHOST ASAP.

I am slightly inconvenienced by this.

@El papa John

Imaginate, y yo como vivo en Francia, el banco me descuenta algunos céntimos de Euro por cada transacción con DH, o sea que cuando viene a ver salgo perdiendo…

How are we supposed to run a reliable business if dreamhost isnt reliable ! What do I do? My web guy is o/s and I am in AUS without a website trying to run my company

Please Dreamhost, faster..faster…we depend on you…

Thanks for the steady updates, only 1/2 of my sites are down, but my VPS that hosts source control is down which really hurts us.

This is ridiculous. The sites have now been offline for 6 hours. How are we supposed to run websites on servers like this? Are you going to compensate all of the people losing money?

@ El papa John

A no ser que lo descuenten en días con respecto a la fecha de vencimiento del contrato, o sea que si tu contrato se vence el día 14, se venza el 20.

– Bluehost??? No se vayan para BlueHost!

March 19th, 2013 at 7:10 pmNightmareHost Says:

@Nikki J – wow, that’s really lame that you have to put in a ticket to get credit.

March 19th, 2013 at 7:10 pmEl papa John Says:

@Guerrero

No, porque ellos te hacen el descuento en tu próxima factura, no te reembolsan nada. Así que si sigues con ellos pagarías menos.

Lo bueno es que en francia es la madrugada y acá en américa es de tarde así que baja el tráfico a los sitios.

Está muy duro este fallo.

March 19th, 2013 at 7:13 pmEl papa John Says:

héchale webos pinche dreamhost que esté mañana temprano, sino mi abue me deshereda. Y se levanta con el gallo la condenada viejita.

March 19th, 2013 at 7:14 pmI_Pay_For_Traffic Says:

Would have been nice to send out an email letting everyone know what’s going on…. especially people who pay for traffic. I’ve lost so much money today because of this. I can’t afford to have this happen again. I have to move everything to a more reliable hosting company. FFS!

@ El papa John

Bueno por un lado, pues 3 de mis sitios están dirigidos a la comunidad hispana, y la mayor parte de ella está del otro lado del Atlántico… Tengo un solo sitio en francés que no está del todo funcional.

Pero bueno, mis sitios son poco conocidos por el momento y no son sitios para ganarme la vida con ellos, sino mas bien informativos y proyectos tipo Wikipedia…

My website is down! Place hosting is: US – West (Los Angeles, CA) I’m very upset with this…

March 19th, 2013 at 7:17 pmRudy Simpkins Says:

Это очень плохо!! когда мы можем ожидать, что это фиксированная?? Все мои сайты вниз, и это большой день запуска!!

“We are in the process of deploying a spare to that location and we estimate that will take 1-2 hours…”

Drive time from LA to Irvine.

I still don’t understand why there’s not a backup to the backup to the backup to the backup. And in different locations, not on the same grid. This kind of problem should be anticipated and prepared for. We have been offline for about six hours in the Midwest. And yes, things like search engine ranking is going down the tubes among other things. My competitors don’t have this problem!

Yo voy a dormir.

Esperando que se resuelva el asunto y que no haya pérdida de datos.

Animo al equipo de DH y paciencia a los más afectados: “con paciencia y calma se sube un burro en una palma”.

Dear Dreamhost,

Please let me know what server you use for dreamhoststatus.com. I have never seen it down.

Thank you

March 19th, 2013 at 7:20 pmEl papa John Says:

@Guerrero pues mete el ticket, no pasa que te digan que no, o te dan una lana, pues te comes una chapata o una botellita de agua en salud de dreamhost, yo ya metí el tiket igual para que andan de hocicones.

Dear DreamHost: at least change our dead websites’ “Error Establishing Database Connection” to “DreamHost apologizes for being so utterly unprepared for a disaster. It is not the fault of your webmasters that services are not available. Again, we apologize for being so CRAP at our jobs.”

COÑO!!!! no es relajo estoy perdiendo dinero coño!!!!! hasta cuando yo pago por este servicio, y pahgo siempre al día, exijo que arreglen esto ya!! y exijo que me descuenten o me reembolsen mi dinero!! Simon Anderson ya te estas pasando!!! es mi plato de comida el que está en juego no el tuyo quiero respuestas!

Wow, I’ve been hosting with them less than a year and had downtime on a monthly basis. I thought I had a mediocre host before this, and they had no where NEAR the down time I’ve had with Dream Host. Regretting making the switch right now.

Slightly inconvenienced by the site going down, but I’m really disappointed in the people who are commenting like lunatics. Is this what counts as something going seriously wrong in your life? Wow, I’d had to see what happens when you run into a real world problem. For crying out loud, grow up and have a little patience.

Dear Dreamhost,
Please let me know what server you use for dreamhoststatus.com. I have never seen it down.
Thank you

=> http://www.cirtexhosting.com/ =<

This is ridiculous… I’m done. Completely unacceptable for a major hosting company!

EVERYBODY JUST CALM DOON

I wish this had happened on Sunday when I was signing up.

Now is up

=]

March 19th, 2013 at 7:26 pmEl papa John Says:

Todos se caén, el chiste es en cuanto se levantan, ahora si se tardaron estos compas.

Oops! Google Chrome could not connect

Not is up to me!

I’m down!!! :(

I’m losing money!!!

My website is down! Place hosting is: US – West (Los Angeles, CA) I’m very upset about it!

March 19th, 2013 at 7:27 pmSteven Savage Says:

To be fair to dreamhost, a bad power outage can do some serious damage, especially since most failovers don’t really get tested until real life walks in. I’m sympathetic, I’ve been there.

But, seriously, this was badly handled and cost a lot of people. I hope Dreamhost takes appropriate measures, since I’ve had great results since I used them.

DREAMHOST DEDICATED SERVER STIPULATIONS

2. DreamHost is under no obligation to compensate Customer for downtime, whether the downtime be caused by Customer, DreamHost Web Hosting, or DreamHost’s upstream providers.

SHARED BENEFITS

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

Well, I’m joining the queue on this one.

I have signed my work up to you guys and now I am copping it big time.

Consider this me cashing in my 100% uptime.

Please fix this, that’s all I can ask. I don’t have to scream at you, your equity holders are already on the case, I’m sure.

That’s IT. Never using your crappy service again, and I’m telling all my clients the same. The inconsistency in service over the last YEAR has been totally unacceptable. I was a big proponent of yours, DreamHost, but I have to say… you suck now. Much more trouble than you’re worth, even as a ‘value’ host.

11:55pm pacific time means 23:55 (now 19:23) = 4 hours and half more of outage

crapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcra

Still not up for me. Not amused.

You mean nothing is up? :) I guess I am getting out of business because of that… I am delivering online service. Nobody is going to treat me serious now. I got a lot of angry emails…

TRY BEING IN AUSTRALIA…. they timed it perfectly.. about 8am til 12 noon … half a business day gone.. 30 staff in office unable to work. Have been DH customer for over 10 years.. same as the last outage… shared hosting all fine… older accounts fine… but NEW VPS DOWN. last outage… shared hosting was not effected only VPS ….. the client who pay the most get the worst service… crazy … makes me think… why pay $200 per month for VPS …. is it really faster than shared hosting… its certainly not as reliable

Wow… sounds epic… I’ve only had DH for a few months. I really hope it’s not a regular thing. The comments are not encouraging at all though. :( I was looking to pay ahead a year or two in a few months time…

March 19th, 2013 at 7:31 pmRecommendations pleas Says:

SERIOUSLY – SHOPPING AROUND. RECOMMENDATIONS WELCOME.

Bluehost?
Hostgator?
any others?

crapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcrapcra

Just think, somewhere out there, there is someone who would have been on the web all this time. Instead, maybe they sent for a walk, or called their mother. Nice.

assssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

Update March 19th 7:19pm:Our data center provider is also initiating emergency maintenance on the UPS system that caused the initial power outage. This maintenance is scheduled to be complete at 11:55pm pacific time. We don’t anticipate problems to stem from this, but are bracing for issues, should they pop up.

11:55pm THAT MEANS ANOTHER 3 HOURS ??!

Dreamhost we depend on you…

My website is up, but is also very slow and is not working fine

Our conference site is hosted with your service. We ran articles in major publications today that reached tens of thousands of readers. Our single intent was to drive traffic to our site for increasing registrations.

I understand that things go wrong, and sometimes many, many things go wrong, but at this point, we are looking at a potential loss in conference revenue. I do not thing this many hours of down time is providing a reasonable service. I’d be happy to discuss this further, if and when you have things running again, so that you can help me understand how this was unavoidable.

I do appreciate the detailed updates and transparency, but really all we need is an estimated time that you’ll have services back to normal. All the additional detail is not relevant to our business – we just want to be able to tell our customers when we can expect to provide them with the information and services they need.

Thank you for your valuable time.

Still waiting for my website to up… dreamhost is totally crap. One of my friends websites are up but for me no luck till now.

this a fucking joke?

Server pulcherrima is down about many hours!!!!GOD!!!

@Recommendations: I read Westhost was good if you use wordpress. I also read hostgator is crap and they ad (not sure if they still do) advertisement from them on your site in the code. Maybe to get their rank up. Plus they pay more to referrals than the cost of their service, so it sounds shady to me. I would research more on it maybe.

Can you confirm that the previous statement is true that there is no data loss. Please confirm again. THanks

Im on a VPS but was literately 2 days away from migrating all my other sites over from cirtex and getting my own box… But this is sketchy to say the least,,,

“Our 100% uptime guarantee means we take uptime seriously.”

I don’t know about the guarantee though. Saying you get a free day for every hour of downtime is like saying, “if you don’t like our ice cream, we’ll give you a gallon of it for free.”

March 19th, 2013 at 7:45 pmsomeannoyedperson Says:

LOL @ chris posting http://thewebsiteisdown.com/ I’m watching episode 1 and I’m laughing so hard I kind of don’t care about the down time right now. Thing is, I couldn’t watch them through the site. Then I thought maybe the site is purposely broken to fool us because it’s “TheWebsiteIsDown” so then I went and searched for them on YouTube and now I probably won’t do any work until I watch all of them. Thanks.

All of my sites are still down.

March 19th, 2013 at 7:49 pmTrinity Sales Says:

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

Not impressed ! I am losing money!!!!!!

Dreamhost never again!

I want my money back.

Weeee! todas mis páginas estan arriba, todo se arreglo y no perdi nada nada de dinero!!!!

MENTIRA!!!!! estoy perdiendo dinero como un loco y nada se a arreglado!!! exijo una solucion ya!!!!

Come on the 100% guarantee gives you 1 day for every hour, which is worth like 30 cents for an hour! So “From a business perspective this is crazy!”, definitely!

I hope the new Pope, Francisco, help you to give us a better service.

@JL “Come on the 100% guarantee gives you 1 day for every hour, which is worth like 30 cents for an hour! So “From a business perspective this is crazy!”, definitely!”

INDEED! I want real money!!!

What about an extension cord.

I have to wait for another 4 hours? ummmmmmmmmmmm

this is sooooooooooo frustrating! can’t wait to move over to hostgator. come on dream host? WHYYYYYYYYYYY????????? and no i don’t want the reason you are currently experiencing issues! i want to know why you don’t have some kind of a back up plan!?

my friend uses host gator for 10 years and had ONE problem. ONE!

i used dreamhost for a couple of years 3 years or so ago and NEVER had a problem. this time around i’ve had SEVERAL issues in 8 months.

my only guess is you WANT people to go somewhere else.

“The DreamHost 100% Uptime Guarantee!”

“The DreamHost 100% Uptime Guarantee!” jajaja

I think some of you people are out of line and just being rude. Get a grip. Most of you probably pay $10 a month for your shared hosting plan. What would normally take you hours to do, if you even new how to, DreamHost simplifies for a great price. I am sure they will learn from this and the service will be better because of it. Bad timing… yes. Frustrating ….yes …. Lost revenue…. maybe….. end of the world… I think not. Get off your computer and spend time with your family. There are people out there right now with real problems. Daniel Barden was my cousin. Take the message of this page to heart and chill people. http://www.facebook.com/whatwoulddanieldo

WWDD, you must be new here.

March 19th, 2013 at 8:02 pmPissed Off Consumer Says:

They mean that http://www.dreamhoststatus.com has 100% uptime when the power goes out :)

Who is Daniel Barden?

I can’t help but wonder how many people complaining here are PR people from other companies. I’m paying for a 160 dollar dedicated on DH, but I’m just waiting patiently.

I get hosting clients from DreamHost all the time. Now I see why.

This is Horrible experience. Very poor service and reliability.

Daniel Barden is my cousin who was killed in the Sandy Hook school shooting. 4 months ago, this would have made my blood boil as well. I have over 50 websites down right now. Bottom line is… I can’t do anything about it so I am going to use this board and audience to promote something good rather than rant about how !@#$%^ DH is.

“I can’t help but wonder how many people complaining here are PR people from other companies. I’m paying for a 160 dollar dedicated on DH, but I’m just waiting patiently.”

You wouldn’t be waiting patiently if you spent 10x as much and were losing $200+ per hour of downtime.

March 19th, 2013 at 8:07 pmConsciousness Says:

I don’t see why you don’t just go ahead and destroy this data center. All it does is cause problems.

March 19th, 2013 at 8:08 pmPissed Off Consumer Says:

The reason dreamhoststatus.com is up and running is because it’s hosted with http://www.linode.com/ :) that is too funny.

i guess we are all staying up today

March 19th, 2013 at 8:09 pmProfit from Tragedy Says:

The American Way.

“On Wednesday, March 20th, we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption….”

Do ya get it now. Switchover failed.

redundancy nope! I have just had two customers say that is it .. bummer — does anyone know of a more reliable service for under $200 a month ? I have 120 clients let me know please.

Dreamhost says:
“we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee.”

Note that 10% part. 10% of 30 days is 3 days…which means if we’re down over 3 hours (which we are now into our 6th hour), we see NOTHING MORE. DREAMHOST SUCKS!

Sorry for your loss, WWDD. Many many prayers.

Daniel Barden is my cousin who was killed in the Sandy Hook school shooting. 4 months ago, this would have made my blood boil as well. I have over 50 websites down right now. Bottom line is… I can’t do anything about it so I am going to use this board and audience to promote something good rather than rant about how !@#$%^ DH is. http://www.facebook.com/whatwoulddanieldo

Down in Australia

March 19th, 2013 at 8:12 pmPissed Off Consumer Says:

@derpderp, Does it really matter how much of a swinging dick you are with you $160 package? Every person on here is still being impacted by a less than 100% uptime SLA. This fuckin sucks for everyone who has stuff hosted through these guys and it doesn’t make your site *more* important. Maybe to them you are more important because you pay 160 bucks, but I am sure they have 100’s of you too. We are all just a number either way.

Exacto! Patrick Delahanty

“@derpderp, Does it really matter how much of a swinging dick you are with you $160 package? Every person on here is still being impacted by a less than 100% uptime SLA. This fuckin sucks for everyone who has stuff hosted through these guys and it doesn’t make your site *more* important. Maybe to them you are more important because you pay 160 bucks, but I am sure they have 100′s of you too. We are all just a number either way.”

I was quoting that other guy, dude.

#######################################################################################################################

Dreamhost: “We are still getting the new network device in place and configured. We’re doing everything we can to work as fast as possible and restore service ASAP.”

So you don’t have any backup / spare device? :-D

#######################################################################################################################

March 19th, 2013 at 8:14 pmPissed Off Consumer Says:

Patrick Delahanty your 100% right.

Please do me a favor. Instead of talking about stupid stuff in your newsletter, talk about how you’ve screwed your customers twice in the last 6 months and how you’re going to fix these problems.

This is an unacceptably long time to recover!

“I can’t help but wonder how many people complaining here are PR people from other companies. I’m paying for a 160 dollar dedicated on DH, but I’m just waiting patiently.”

I can’t help but wonder how many people here trying to protect DH reputation are from DH PR.

March 19th, 2013 at 8:15 pmPissed Off Consumer Says:

Soon they are going to say they have to get some fiber and cat5 because of a cable shortage.

My sites are not even at this data center, they are at US – West (Los Angeles, CA) but they are also down. How about an update on that issue?

I would probably have better luck hosting my sites at http://www.nightmarehost.com

I guess I am not renewing the service. A 5 hour outage is not acceptable in anyway.
You don’t have a spare on site?
Stop crying about losing money. It’s your own fault to rely your business on Dreamhost and don’t have a redundant site. Don’t you know that Dreamhost gets power/network/storage issues every now and then?

ARE YOU SERIOUS?!??!??!? ALL MY WEBSITES ARE DOWN AGAIN!??!?!

How can this webhost suck so much???

This is unacceptable.

I don’t think that the network improvement series is working all that well…

Read everything that happened up above and it seems like everything plus the backup failed. All of you need to take a chill pill. Life goes on.

please hurry !!!!!sos!!!!

Several of my web sites are still down.

All of you need to learn to build your own redundancy.

30 minutos pude pasar, 1 hora hasta en los mejores proveedores pasa, pero 6 horas!!!!! que clase de equipos tienen en dreamhost ? no se si estoy exagerando o que, pero 6 horas!!!!!

si de aqui a mañana eso no se arregla, es definitivo uqe me voy para otro proveedor!

esto es el ridiculo.

Shit happens guys, yea it sucks but when you have two data centers get fried there isn’t much you can do.. will take the night off and enjoy reliable service tomorrow.

How about my data???

hundreds of people who normally livechat while listening to a 9-12PM EDT tuesday international radio show are pissed that friendsoftom.com is down during the show. geniuses.

This is unacceptable. I am loosing 1 trillion dollars every second we are down. ONE TRILLION DOLLAR!!! I demand to be paid back! Let’s see… 1 trillion dollar/second times 7 hours = huh ho… my calculator’s power source exploded.

Simon Anderson manda a comprar equipos buenos! deja de ser tan duro y compra ups. No seas tacaño!

March 19th, 2013 at 8:34 pmFred Flinstone Says:

Yabba Dabba Doooooo!

You just read that in my voice.

WWDD people WWDD. Some people have real life problems. http://www.facebook.com/whatwoulddanieldo

Update March 19th 8:30pm: We’re having some success resurrecting part of the problem areas of our network. This does not mean hosts are up just yet, but it does mean we’re getting closer.

———–
WTF ?!?!?!?!

como minimo deberian dar 2 meses gratis para todos, creo que es justo.

@OK Says: they sound pretty desperate :)

How in the heck can you have so many outages? I’ve had my site with you for 3 months because my new developer said he was familiar with your service and preferred I move from my old provider. In 3 months my site has been down more than it was with BlueHost in 5 years. How does this happen so often to a company that has an About Us page that says “Our technology is built by some of the industry’s most clever developers on stable systems that come together to create one amazing user experience. Designed to be easy-to-use, everything at DreamHost is built with the customer in mind.

So far, we’ve hit the nail on the head. Over the last decade or so, we’ve grown to more than 1500 servers, state-of-the-art data centers, and a full time staff of more than 100 employees. We now host more than 1 million domains. We’re focused solely on providing a quality customer experience backed by the absolute best web hosting technology in the industry. It’s a never-ending quest, but it’s one we’re happy to take on.

We pledge to always take your business seriously. We’re always looking for ways to diversify our offerings, improve reliability, and improve our overall quality of service. After all, your ideas put you online, but we’re the ones that help keep you there.”

y xinga tu madre tambien :)

hey we can chat here O:-)

Obligatory update every half hour with no real update. Yay.

i suggest rebranding:

reamhost.com

simple change!

jajajajajajaja

######
Update March 19th 8:30pm: We’re having some success resurrecting part of the problem areas of our network. This does not mean hosts are up just yet, but it does mean we’re getting closer.
#####

Esto tiene que ser mentira!!! que clase de comunicado es este ???

es loucura, mi amigo

CAN I LOST ALL MY DATES???

CAN I LOST ALL MY DATES???

CAN I LOST ALL MY DATES???

CAN I LOST ALL MY DATES???

all your site are belong to crash

So the UPS BACK UP system is what has caused the outage?? thats a bit of an oxymoron isn’t ??

Waiting for services to go back online so I can work… this is a great substitute though…

I may turn this comment log into a play.

#easilyentertained

@Jason That’s called marketing. Nothing to do with reality.

Time and money are losing

@fffalcon: they richly deserve some comedic ridicule

http://www.dreamhoststatus.com is always up and hosted on http://www.linode.com, so should probably look at it :D

Relax guys, there is no power outage that lasts for hours and hours, its something else. This is the third time I go through this type of problems with DH. What I recommend is to have your latest software and data back-ups ready, and I hope you have them, because from my experience, DH will do you a nice favor and restore all your stuff but it will be at least 2 months old. So, don’t forget to check your site is not serving your Christmas specials when it comes up :)

March 19th, 2013 at 8:48 pmHomer Simpson Says:

Guess we shouldn’t have hosted our reactor management software here…whoops.

Kind of funny that my site in LA are fine but my clients sites in VA won’t load.

Let’s see if we can get this thread at more than 1,000 comments by the time DH restores service. It happened last year: http://www.dreamhoststatus.com/2012/01/29/web-ssh-and-ftp-services-for-a-subset-of-vps-shared-and-dedicated-machines-down/

dreamhoststatus is hosted on http://www.linode.com and is always up, so you should probably check that out :D

March 19th, 2013 at 8:50 pmPissed Off Consumer Says:

WWDD, sorry to see you are down such a noble site and sorry for your loss.

hurry1!!!!

Hi, any updates? I still can’t acces my account. Please solve the problem as soon as possible. Thanks

Our site is down also. DreamHost what is the recovery eta?

completely down, but god what a refresh this is to know that i cannot do anything but wait. Maybe ill take my bike out for a spin, or the dog for a walk. Im hoping that Dreamhost will compensate everyone who actually utilizes their hosting for this, but lets get real. Couple hours down, lets party

###########################################################################################################################################

Give us 1 month free hosting

###########################################################################################################################################

Update March 19th 8:46pm: Unfortunately, we have now determined that our sole remaining data backup is on Uranus, and the rescue vehicle will take some months to get there and back to Earth again. Enjoy your bankruptcy, customers! :P

################################################################################################################################################
################################################################################################################################################

Compensation: 1 Month FREE Hosting.

################################################################################################################################################
################################################################################################################################################

March 19th, 2013 at 8:54 pmPissed Off Consumer Says:

They have been down so long I have managed to read all volumes of Donald Knuth, The Art of Computer Programming. http://www.amazon.com/Computer-Programming-Volumes-1-4A-Boxed/dp/0321751043

If only my gf could go down this well.

March 19th, 2013 at 8:54 pmHomer Simpson Says:

It’s 8:54PM. Do you know where your packets are?

my #1 client just dropped me, dreamhost I hate you and I think i’m not the only one. I hope nobody is going to trust you anymore and leave you like I will do. No power backup for a business that big that so many people depend on… you are messing with to many peolple’s money here, hope your company crash.

@Homer: Lost in Space

March 19th, 2013 at 8:56 pmPissed Off Consumer Says:

G D, hahaha that’s fuckin great.

Most productive thing all night is reading this comment board.

Um … you fried TWO network devices? WTF? Don’t you have power conditioners?

March 19th, 2013 at 9:00 pmPissed Off Consumer Says:

Bobby, I left something as productive as dreamhost in my toilet a few minutes ago.

My website is finally up.

March 19th, 2013 at 9:01 pmPissed Off Consumer Says:

Power conditioners? DH replies… Are those found in the hair care aisle?

Dreamhost you are ensuring that I remain in poverty, every time I have 10 clients about to sign something, my site goes down, this is really, really not cool.

March 19th, 2013 at 9:02 pmDiogo Medeiros Says:

up!

Still our sites or downside….yours not resolved this issue.

Please resolve this issues and also don’t repeat this issues.

Article posted on Sept 16 2012

DreamHost was down for most of September 12th because of the blackout in Los Angeles, where their data center is located. Their backup generators failed. This is the message on their support site today:

Currently, we’re having a new issue with our core router dropping a lot of packets, which is causing severe network latency for a number of our customers. We’re actively looking into this now, preliminarily it looks like one of the two (redundant) CPUs may be dying, quite possibly related to the power outage yesterday.

Posted here http://www.upstartblogger.com/why-dreamhost-sucks

So it happened before?????

Site is still down and can’t login to wordpress.

You guys expect a huge host service to have actual secure and totally backed up systems? Hmm, that would make this site *dreamers*host.com

13 Years with Dreamhost. Too bad I have to leave like this. But I have a 4.5star app with 20k downloads that hits my website for instructions. No more excuses or giving Dreamhost second, third, fourth chances. Time to leave. Quality just is nothing like it was in 2000.

still down .no up!!

@Dan you are right. Last week DH helped me update MySQL version on my PS. And restored (why?) an old version of DB. Took me 3 days to realize it and it was too late to grab my own backup.

yep, no up, i check the site i am trying to use

I’m at the point I can’t refer customers to DreamHost anymore and I’ll start to port them to a new host! It cost me more in time and lost money when my clients call me with hosting issues that I have to troubleshoot the problem. The majority of the time its because the websites are slow and the support is horrible. Downtime is also more than most hosts I’ve dealt with and today’s 5.5 hours and counting is unacceptable.

Good Bye DreamHost!!!

Does it really matter if I put my website here on the comments? It’s fucking down anyway. But this is how I see it, there was a planned power outage and either DreamHost didn’t choose to do anything about it or their ‘backup’ system in place had failed. So what I want to know is, what the fuck happened? I don’t give a shit about what you’re doing right now every minute fixing it, I want to know why this happened and/or an ETA.

You can hide behind your million little minute by minute updates to make it seem like you guys are all over this shit, but the actual truth is, you’ve fucked up and in order to make yourself seem that little more reputable (only just by a small margin) you post a whole shit load of little incremental updates that make people go and read and think “ahh they are all over it and fixing this shit” – get real. Your customers just want a fucking ETA, so WTF is the ETA?

plase hurryyyyyyyyyyyyyyyyyy

March 19th, 2013 at 9:08 pmPissed Off Consumer Says:

Eric K, sadly after trying about 35 different hosts out there… dreamhost’s features were second to none from shared to VPS etc.. They even have ssh access for all tiers of service which is something that is not found. I personally hate CPANEL’s subdomain for addon-domain crap.

free ads ahuevo!!!

I can see some of my site are up

Also, I just noticed that dreamhoststatus.com runs WP 3.3.1 (precisely 14 months old today). So much for asking us to keep our crap up to date.

THERE’S HOPE, AFTER 7h DOWN MY SITE IS BACK UP, LETS HOPE IT LASTS

Now is up and all is working fine.

This is unacceptable. Unbelievable that this has happened 2 times in two months.

Our customers are pissed. What is DreamHost doing to reimburse people for lost income, lost clients and lost sleep.

It’s been almost 7 hrs!!!

Unbelievable for a hosting provider of this size.

Han Solo warned us. You can’t just pick up a spare thermal coupler at the 7-eleven.

赶紧的,太墨迹了。你们搞毛呢。

They must have misplaced the Illudium Q-36 Explosive Space Modulator.

Done… During this (yet another) downtime w/ DreamHost, I have moved 2 of my domains to BlueHost. 4 to go…

Every time I’ve added a new customer to DreamHost, they’ve gone down that same week.

still down damnit. really need to find a reputable hosting company.

all down,no up!

I have had pretty good service from Dreamhost, but losing customers and access to my shit is really bad. Hope they take care of me for this.

Hey, finally got a ping at 9:10pm PST !!!

No website though :(

This 24-hour Network Improvement Series Maintenance is just awesome.

Why worry about Time-To-First-Byte right? :-P

My 3 year hosting plan expires & needs to be renewed in April 2013. Which is perfect because it gives me a couple weeks to transfer everything elsewhere.

This is insane! Even their best service dedicated server hosting …. wtf

Guys we are just getting frustrated. DH guys are not bothered. Probably not even reading this. Probably they are just laughing off reading these messages.

Businesses feed on human psychology.

i wanna play pokemonshowdown.com and it’s down!! where do i even get my pokemon FIX?!?!? plz hurry you guys :(

Update March 19th 9:15pm: We are sorry for the inconvenience, but we are having gaming night tonight at the office with the guys and didn’t find any free power sockets to plug in our PS3, so we decided to unplug the power socket for the servers. Thank you for your patience!

Comments posted here may not be viewed by DreamHost staff at all.

Is Dreamhost a trusted company? They did not informed me about this disturp

lol, amateur hour

This is the longest server downtime in my 5 year online business. Definitely there is very big problem there. I hope you guys are trying your best to solve the issue.

Hey guys,

This is my opinion on this whole thing: Its pissing everyone off, really bad.

So im thinking we will all need a good craft beer when this is all said and done. And maybe, just maybe we can look back and laugh at this one day.

Because this is the day you discovered that there is a Hip Hop Beer Festival in this crazy world…yes, HIP HOP BEER FEST. The first and only Multicultural Craft Beer Festival company in the world is bubbling out of Atlanta. And soon we will be at a city near you.

So stay calm and drink beer…good beer. And when this situation is settled and we are all making a million dollars a day from our sites, you can go check out HighGravityHipHop.com and feel better about the world we live in…because there are actually people out there trying to unite it.

Peace,

Lenox
President & Founder

PS Yes, I just did that…

I am quite sure the quality of service is alot more important than the planned gaming night…and you don’t play PS3 in a server room, either…

Update March 19th 9:15pm: We’re still working away on this, and are now seeing some services restored. You may see some intermittent outages for a bit while we work out the kinks, but services are coming back online slowly.

Time to move to Amazon cloud.

AWS.AMAZON.COM

March 19th, 2013 at 9:31 pmPissed Off Consumer Says:

“intermittent outages” for a bit while we “work out the kinks” … I’ve got a couple of kinks in my pants they can iron out.

I’m moving all my domains off Dreamhost.

This is very bad, its taking such long time and effecting business of thousands of users. There should be alternative and effective way to avoid such situation. I would never prefer to use this hosting company ever from now.

You wanted an ETA?

Not a problem.

Everybody Try Amazon.

March 19th, 2013 at 9:33 pmPissed Off Consumer Says:

Suresh, AWS sucks cocks too don’t think they all that great. The pricing model for EC2 and custom images are a pain in the ass especially Linux. I have spent too much of my worl life working in AWS and it’s total crap IMHO.

guys,

this is a major issue. and something that convinces me I should be moving across to bluehost.

does anyone have any experience with their server and know if they handle this any better?

also – why doesn’t my ‘redirect’ setting pickup? trying to redirect my users to a http://www.blogger.com “server down” page but no update from dreamhost to handle that yet…??!?!?!?

BROKEN.

March 19th, 2013 at 9:33 pmone of DH user Says:

>BOTH devices were fried

fried?
I wonder they don’t have those thing that protect devices from power surge?

With 99% uptime, they’re allowed 3.65 days off a year. :(

EPIC

FAIL

LULZ

the least dreamhost could’ve done is notify me after my website was down after 30 minutes?!?!?!?!? fk

Power’s off for 6 minutes, takes 6 hours to fix. Nice job.

@K: Dreamhost got your monthly payment. They are doing just fine, thanks. No sweat.

Guys, they are trying…

I mean, do you really expect two routers/main network devices to fry in one go??

Why di BOMBOCLAAT mi fucking site dem nah work? Man pay dem pussyclaat money fi service an a dem fuckery yah unnuh a gwaan wid. A weh di bloodclaat unnu really a deal wid dreamhost? A dem bloodclaat yah mek man fuck off wid unnuh unuh. How u fi have a fucking 5 hour bomboclaat outage? Yow fix di raasclaat sitn before mi haffi tell di hol a unnu fi go suck all a unnu madda dem!

Some of my shared hosting sites are up now. :)

Ted

I’ll have what Ras is having. :)

The server at psXXXXXX.dreamhostps.com is taking too long to respond.

:(

-Jamie M.

How this can happend to a major Data Center?! Power Outage?!
I mean, that’s why I host on an external provider: For redundancy on data, and infrastructure (yes, that includes power!!!)

This is not the first time I have my servers down. A few weeks ago, the storage device for Banjul went down and down-time was critical.

I love the fancy and cool newsletter I receive every month but I can’t work with a hosting provider that have so many incidents!

Time to move to another provider?

Almost up to 7 hours now. Free week of hosting, woohoo! The revenue lost from all this downtime could buy us a good amount of real hosting.

March 19th, 2013 at 9:40 pmPissed Off Consumer Says:

I think what boggles my fucking mind most is that every data center I worked in had both battery banks and diesel backup generators for long term power. They were configured to fire up once a month to ensure everything worked. We had Linux boxes that had over 3 years uptime. How can a company as large as DH fail to test their power systems redundancy?

This is what i am going to do while I have time on my hands.

I will include the URL of this page in my sitemap.xml, and I will do it for every site I have. Then, I am going to my Google and Bing Webmaster accounts and resubmit all the sitemaps.

@Pissed. Oops :)

Thanks for keeping us updated!

March 19th, 2013 at 9:43 pmone of DH user Says:

This reminds me that 6 or 7 years ago their data center in Florida caught fire in middle summer day.
And it never come back for couple of days after that.

My WP blogs are still down :-(

“ThomasM Says:

Guys, they are trying…

I mean, do you really expect two routers/main network devices to fry in one go??”

Well it is 2013. So…yes.

March 19th, 2013 at 9:44 pmEl papa John Says:

No entiendo como chingados, con los servers apagados, mi vps sigue contando el uso de la memoria. Esa tecnología de punta me asombra.

The long-awaited zombie apocalypse has begun.

And dreamhost is the zombie.

March 19th, 2013 at 9:45 pmPissed Off Consumer Says:

@G D, yeah I guess so. SNARF! Did anyone call the oil company to re-fuel those generators out back?

My site is still down. I am someone that has used dreamhost for ages, recommends it regularly, but will cease to do so on all fronts. total bs.

you all should be ashamed of yourselves.

Dreamhost. Because it’s only just a dream?

So… if this is just an Irvine datacenter outage… why are all of my sites, hosted on Gwinnett (east coast datacenter in Ashburn, VA) down, then?

El papa John ahahhahaha asombroso verdad ? jajajaja

It’s unbelievable how DH reputation was thrown to the toilet in just 6 mins.
What they will do to compensate this?

March 19th, 2013 at 9:50 pmPissed Off Consumer Says:

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Your site is fucked.
Your site is fucked.
Your site is fucked.
Your site is fucked.
Your site is fucked.
Go fuck yourself.
Go fuck yourself.
Go fuck yourself.
Go fuck yourself.
Go fuck yourself.
You paid for this.
You paid for this.
You paid for this.
You paid for this.
You paid for this.
Ha Ha Ha Ha Ha.
Ha Ha Ha Ha Ha.

Dreamhost. As in dream that your business is not really suffering.

All of our DREAMS spoiled. :(

Because you mustn’t trust master. He deceives us.

(Yes, I’m watching lotr while dreamhost tries to figure out how these so called “computers” work)

March 19th, 2013 at 9:51 pmone of DH user Says:

FlavoredAir,

as far as I know,
william-hooper.dreamhost.com has no problem.

Mine around 70 websites are still down :-(

Extremely unhappy with the dreamhost. This type of long breakdowns are not expected from dreamhost.

All of my sites are still down. So many opportunities lost today. :( Such a bad day.

March 19th, 2013 at 9:52 pmPissed Off Consumer Says:

I have a dream, that all web servers will have 100% uptime.
I have a dream, that all hosting companies will be judged by the content of quality of their services, and not their failed SLAs.

Still nothing.

###############
Update March 19th 9:35pm: Ok, we see a lot more sites and servers coming online, so this is good news! We’re still monitoring things, to ensure they come back online smoothly.
###############

MENTIRA!!!!

After nearly 7 hours, only half of my websites are back up – one of them incredibly slowly, at that. I sincerely hope, with the extent of outages in the last 6 months, paying customers will be credited some portion of their monthly fees, b/c this is kind of ridic. I’ve held out through outage after outage since January of last year and I’m about at my wits’ end.

Very ready to take my business to another provider unless Dreamhost offers some kind of compensation for this kind of downtime.

As professor Farnsworth might put it:

I DON’T WANT TO LIVE ON THIS HOSTING SERVICE ANY MORE

Hey, somebody photoshop it:

http://bit.ly/yoeK7L

I think dreamhost will refund for “One Month”. This is solve all problems?

#internetfordummies

When services are ready? my clients are already desperate for not having their sites,

@GC: Don’t hold your breath, dude.

Really? Now I don’t even get the ping I got 50 minutes ago…

Still nothing :(hggggggggggggggg

Please continue to hold while we disserve other customers. Thank you, and listen to some more muzak.

March 19th, 2013 at 9:56 pmEl papa John Says:

@murray si es mentira, que sitios arriba ni que la chingada.

Yo ya me quiero dormir sin pendientes.

March 19th, 2013 at 9:57 pmPissed Off Consumer Says:

The only thing more painful than waiting for my sites to come up would be a midget speed bagging my nuts.

How many languages do they have people pissed in on this board now? :)

[sings] We are the world…

Can anybody confirm anything back online?

We are ready to take our business to another provider unless Dreamhost offers some kind of compensation for this kind of downtime.

Maybe they are trying to run the new SimCity on their servers.

March 19th, 2013 at 9:58 pmPissed Off Consumer Says:

Celso, I come in peace.

Beyond frustrating – and this is effecting our business! This has taken far too long to resolve! Why on earth was there no backup for power?

Isn’t it odd how every Dreamhost Site of the Month from basically 4 years ago (2009) up until February of 2013 is still completely accessible, but my site isn’t. Great…

March 19th, 2013 at 9:59 pmHomer Simpson Says:

Via con queso

me acuerdo cuando dreamhost comenzo cuando estos acontecimientos eran muy extraño y casi nunca pasaban hoy día es otra historia como regla general debe caerse una vez por semana.

como ha decaido dreamhost ya no es lo mismo de antes, se ha vuelto otro proveedor chabacano y basura, estoy casi seguro que ni en los peores proveedores de alojamiento demoran tanto en salir de un problema como este.

Dreamhost can just eat a bag of chicken-fried dicks.

Anyone online yet?

The cake is a lie.

March 19th, 2013 at 10:01 pmPissed Off Consumer Says:

@G D, chicken-fried dicks… mmm crunchy.

Online? What is this “online” you speak of, earthling?

Metallica – Nothing Else Matters [Official Music Video]

http://youtu.be/Tj75Arhq5ho

A ver si llegamos a los 1,000 comentarios.

A webo!!!!

Si se puede! Si se puede!

pokemonshowdown.com :( I WANT TO PLAYYY @_@ but I can’t !!! unacceptable!! I need to start battling some pokemon or I will go CRAZY

Celebrating 15 Years of Web Hosting! – DREAM HOST

@Christina no creas lo que te dicen, ninguna página está arriba, eso es mentira.

My site is still down
Please Francisco I, help me!

I just realized we are dreamhostages.

WOW. 1000+ posts

Pope Francis can’t write you a papal bull, because DREAMHOST HAS ALL THE BULL RIGHT NOW.

March 19th, 2013 at 10:05 pmPissed Off Consumer Says:

@G D, only without the blindfolds and ballgags.

no fix yet? our site is still down? we’re receiving a lot of complaints from our costumers. please advise us what’s going on and whats to expect. ASAP!

I mean who needs the internet really.

is anyone online yet?

Did a reverse lookup and every site on the server a clients site is hosted on seems to be down still… really!

todas las semanas se rompe

Espero nos reembolsen algo no es posible que pierdas dinero por esas fallas, y la seguridad que nos prometieron?

tengo 8 horas esperando y nada, me voy a dormir. Al demonio mi negocio, al demonio mi dinero, ya me canse de esperar

I just switched to another hosting company.

Installed apache, php and mysql from scratch, uploaded my backups (from my site on dreamhost), configured some settings on my server in the time this has been down.

P.S. For anyone who has backups of their site and is running them on another host, the MySQL databases are running SLOW as fuck right now from DH’s end. They’re not down, but they’re causing VERY slow connections. May want to switch over to another MySQL database for the time being.

@HK: Not dreamhost, apparently

My sites are down for yesterday 10::49 and still not up. can you please
tell me that what is happening..?
?????????

March 19th, 2013 at 10:07 pmPissed Off Consumer Says:

@G D, Pope Francis.. Dear load we wait in joyful hope of a web server everlasting until you come again.

I think that if Dreamhost is going to survive, they’d better start airing commercials with big titted girls and cartoon mascots.

@john

Where did you go to?

March 19th, 2013 at 10:08 pmPissed Off Consumer Says:

@Darrell, introducing the Dream-whores

Dreamhost rebranding idea:

Godoody.com

We treat you like turds!

Launch it with a Superbowl ad buy.

March 19th, 2013 at 10:09 pmPissed Off Consumer Says:

@Darrell, introducing the Dream-whores. While your getting fucked, we’re fucking them.

“Dreamhostages”.

Bravo, GD. +3

“G D Says:

I just realized we are dreamhostages.”

hahaha. win.

Now Celebrating 15 Years of Web Hosting! – DREAM HOST

* More than 7hrs Outage
* Still outage continued
* Spoil your Dream :P

How about compensation for the thousands of dollars lost.. not just today, but the last 6 days when we’ve had issues with ATT ISPs… do you guys have insurance to cover our losses?

To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????
To Bluehost or NOT to bluehost?????

##################
Update March 19th 9:35pm: Ok, we see a lot more sites and servers coming online, so this is good news! We’re still monitoring things, to ensure they come back online smoothly.
##################

Still all sites down. Wrong Statement. Cheating Others

@CM:

Dreamhost: “Sorry, our insurance server went down. Oh, and the insurance backup server went down, too. We are making some progress on fixing this issue, but it may take awhile. Or not. We don’t know, honestly. Go away now.”

March 19th, 2013 at 10:15 pmPissed Off Consumer Says:

Don’t go to bluehost it’s jack in the box hosting total and utter crap. Do you research on who owns bluehost and a slew of others. http://www.enduranceinternational.com/endurance/index.html

still down

Never ever experienced a worse hosting company. I’m moving all my business elsewhere first thing tomorrow morning

Wow!
99.9% uptime Guaranteed! This is what my past hosting said. Its a lie!
Now about a month i have transfered here still 99.99% uptime. Yeah right.

Please fix this so that i can start work. Time is GOLD!

Site’s been down for almost 8 hours now. This is a simple power fault?

Claims of backups and redundancy are just that, right?

March 19th, 2013 at 10:17 pmPissed Off Consumer Says:

@MP, a worse hosting company bluehost, fatcow, or even worse ENOM.

March 19th, 2013 at 10:17 pmWebmaster searching for new host Says:

Ok. Lets start posting hosting alternatives. Where are you moving?

Dude, they shoulda bought a Dell.

@Dan

RocketVPS. You have to install everything yourself though. It’s a lot different then DH in terms of there are no “one click installs” etc etc… (But I didn’t use them anyway)

Does not offer “unlimited disk space” or “unlimited data transfer” but my plan is 1000gb data/mo, 30gb disc space, and 1gb RAM. For 11.24$/mo. (And I get a dedicated IP without having to pay for it :))

March 19th, 2013 at 10:20 pmPissed Off Consumer Says:

1% of 365 days is 3.6 days. Dreamhost likes to use the full 1% consecutive.

Also for anyone else looking at RocketVPS: use the promo codes they have on their site. I used Rocket25LIFE for 25% off my billing for life.

Learned a lesson tonight, ran an ad on Facebook for a blog post I’d written…. come back to the computer to a load of responses saying the link doesn’t work… turns out my website is friggin down DreamHosers. Thats $210 down the friggin crapper. Good thing I checked it before it got to my max of $300.

@john

Thanks John. I will check them out. Don’t use any one-clicks either. I’ll probably gone from DH by the weekend.

My sites are still unavailable. What Can I do? or just wait… Please tell me how long would I wait ?

I like Jason’s DreamHosers too

Webmaster searching for new host I’m going back to BlueHost, my developer talked me into DreamHost 3 months ago. Not knowing any different I said “whatever” that was a stupid choice.

We can wait till hell freezes over, is my current guess

My new dream host is my services up another time.

Its down happen 24h after publish we new website and killing the project marketing.

Looking for a dedicated server and suggestions.

if hell freezes over, then at least dreamhost will need less power to keep their servers cool. …their server that aren’t running away. Hey lookit that: win-win.

March 19th, 2013 at 10:25 pmPissed Off Consumer Says:

Finding a new host for the shared hosting folks is going to be difficult if they are used to ssh and sftp access. As I mentioned before bluehost it’s jack in the box hosting and thier parent company owns 99% of the hosting brands that come up in google search. CPANEL shared hosting sucks because they force subdmains when you add and Addon-Domain which I personally hate. CPANEL sucks … I would take plesk over cpanel any day. CPANEL when you are using VPS is ok since you have full access to your VPS and can create real domain entries vs Addon-Domains.

http://www.enduranceinternational.com/endurance/index.html

ok, the site i am pissed about just now came back, maybe after 7-1/2 hours they are managing to get themselves sewed back together

Dreamhost: Please leave your comments and outrage here

https://www.facebook.com/DreamHost

I understand that sh!t happens… BUT it happens to NightmareHost (my nickname for DreamHost) FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR, FAR more than to the other 3 webhost services I use :(

Outage Outrage. Hmm, I like it!

Well… this was a nice surprise to come back to tonight… NOT

Zero revenue today.. hundreds lost.

What is a good host to switch to guys?

March 19th, 2013 at 10:28 pmPissed off Guy Says:

DREAMHOST BETTER OFFER ONE YEAR OF FREE HOSTING TO EVERYONE AFFECTED BY THIS

is rocket vps any good?

still down./………

It’s working! Thanks Dreamhost

We’re in Australia and we’ve been down all day so far, ALL DAY!!! That’s unacceptable. You’d expect some contingency plan in such scenarios. I’ve never experienced an outage of that extend, something seriously wrong happened out there in Irvine… and funny coincident that a day before DH’s move to a new architecture the whole thing goes to the crapper. Will most certainly move to a different hosting solution after that major f-up.

all still down,gosh1!!

March 19th, 2013 at 10:30 pmHomer Simpson Says:

Ping…ping… *silence*

March 19th, 2013 at 10:31 pmHomer Simpson Says:

999

March 19th, 2013 at 10:31 pmHomer Simpson Says:

1000

I demand to be compensated for my account!

My site is still down, this is seriously unacceptable. I need to be able to post new content on the site. How can I do that when the hosting is crapping out on me.

Very bad service from you DREAMHOST.. 14 hours have gone since this gamble. Still my server is completely down. My clients are blasting me like anything as all websites hosted on my dedicated server are of news portals. I don’t know what to do!!!!! Seriously…. I feel ashamed of myself for choosing Dreamhost. Totally unacceptable…

@Alex depends on your needs. If you google the reviews for them you’ll see good talk of them on webhostingtalk.

It’s different from Dreamhost. It’s not for people who don’t know what they’re doing when it comes to managing or running a server. If you don’t know how to install PHP, MySQL, Apache, then I would not switch. Also depends on the needs of your individual site (in terms of data, disc space, etc)

I know how to do all of those things and how to run a server so it’s not a big deal for me.

March 19th, 2013 at 10:34 pmPissed Off Consumer Says:

http://en.wikipedia.org/wiki/Endurance_International_Group

Stay away from these they are all owned by the same company EIG. Fast food of hosting.

2slick.com EntryHost NetFirms
AccountSupport FastDomain NetworksWebHosting
ASmallOrange FatCow Nexx
ApolloHosting FreeYellow PowWeb
BerryInformationSystemsL.L.C. Glob@t PureHost
BizLand Homestead ReadyHosting.com
BlueDomino HostCentric Saba-Pro
Bluehost HostClear SEOHosting
Dollar2Host HostGator SoutheastWeb
Domain.com HostMonster Spry
DomainHost hostwithmenow.com StartLogic
Dot5Hosting HostYourSite.com SuperGreenHosting
Dotster HyperMart USANetHosting
easyCGI IMOutdoors VirtualAvenue
eHost IntuitWebsites VPSLink
NetFirms iPage.com WebHost4Life
NetworksWebHosting IPOWER/iPowerWeb WebstrikeSolutions
Nexx JustHost Xeran
PowWeb MyDomain YourWebHosting
MyResellerHome

Numerous sites still down (Australia)…. unacceptable!

The site and FTP is down. When can I expect it working again ?

Add another one as being in Australia we are experiencing it the worst. It’s almost close of business for the day and it’s still no where near showing signs of recovery at all.

Bad news on the webpage, I couldn’t quench my outrage
I can’t remember if I cried when I saw my website just plain died
But something touched me deep inside, the day that Dreamhost fried

And we were singin’
Bye bye your webhosting is fried
Launched my browser to my website but my website had died
On the comments board we’re drinkin’ whiskey and rye, singing…
This’ll be the day my site died
This’ll be the day my site died…

No update for an hour. Awesome.

So, how long will DH now need to remain up in order to hit 99.9% up after a full day’s outage? How’s that math work out?

@Sweetweaver: No news is bad news :)

My site is still own, 1 hour since last update from dream host, do they think we are all back up? Are they still working on this?

I’m still down. Last time DreamHost had a major outage like this it took nearly a week for them to get everybody back online again. The whole time they kept giving these same updates that implied we were just minutes away.

March 19th, 2013 at 10:41 pmPissed Off Consumer Says:

G D, The three men I admire most is the Father, Son, and a working host.

Maybe we’re all just typing in our urls wrong.

March 19th, 2013 at 10:42 pmPissed Off Consumer Says:

I should have said.. G D, The three men I admire most are the Father, Son, and a working host.

@DownDownDown: Dreamhost is such a visionary company that their minutes are our weeks.

March 19th, 2013 at 10:43 pmPissed Off Consumer Says:

G D, should have said “are” not is… getting sleepy in my timezone.

March 19th, 2013 at 10:43 pmJeremy Wright Says:

down, down, down. Dreamhost, please be honest with us and tell us what is really going on. It is the least you could do at this point.

Ok, to my site says it’s up, but what the heck is “error id: “bad_httpd_conf””??? When I logged in by SSH, all of my data is gone, it’s like a brand new VPS, nothing installed or configured :(

-Jamie M.

Sounds fun, Jamie :)

Another Australian here on their dedicated server.
Been down ALL DAY too. This is probably the longest outage I’ve ever experienced with Dreamhost.

Still down as of 4.39PM EST

At least I would have liked to see an outage page for all the affected domains. Better yet offer some redirect service.

My domain is still unreachable.

March 19th, 2013 at 10:45 pmPissed Off Consumer Says:

I see the late Steve Jobs in his regal black and white photo, with his thumb and index finger pressed against his chin. The caption reads, “Dreamhost… Think Different” fix your shit.

I updated all my pages to return this

This page can’t be displayed

So that when I browse to it, I’m not disappointed anymore.

Still down…any updates?

Think Different? Hell, we’d just take “Think, Period” at this point

still down. we were in transition when this happened so its a big deal, please hurry

March 19th, 2013 at 10:48 pmPissed Off Consumer Says:

Dreamhost the official sponsor of the 75th Bi-Weekly, “Time Out Party!”

Anyone have good experiences using Inmotion or Arvixe for wordpress hosting?

March 19th, 2013 at 10:48 pmJeremy Wright Says:

“We are unable to update the outage page because we are all too busy updating our resumes”…

No manches ya me quiero dormir y no dormiré por la preocupancia.

Another Aussie – our online business has been down all day losing us money.

Paying web developers who haven’t been able to do anything today.

Why, I ask, is Dreamhost.com not down if all our sites are down? Oh, they’ve given themselves plenty of redundancy.

SOBs.

Knock, knock, anyone still here ?

thanks for destroying our businesses Dreamhost

Clients coming to visit my site today, to no avail as it has disappeared. Really hope this gets fixed soon…

Still our sites are down side…..How much time taking for lives on my websites…

this is a joke !!! 6 of my sites are now down for over 7 hours… I am loosing business – clients are complaining… PLS HELP TO SPREAD THE WORD that Dream Host is not ready for mission critical websites…

March 19th, 2013 at 10:52 pmPissed Off Consumer Says:

The SAs at DH will need to create new company names like “Imaginehost – Senior Linux Engineer – 9/2010-3/2013″ … so your that guy.

Still down, seriously??! I’m so glad that I told MANY prospect clients about my website TODAY, yeah that looks really great. losing money. losing clients.

Arvixe has lots of outages too, but they are usually pretty quick and responsive in my experience. My photo gallery is hosted there. Not too shabby, not worse than Dreamhost if that’s any consolation.

Server brazilwood still down!!!

Wasn’t the routing table issue the huge issue that sent many of our sites down for more than 24-hours in 2010? Maybe these very important routers need their own little local UPS system? Just a thought.

I’m so glad that I moved my business websites to Linode and don’t have to deal with these issues.

On the plus side, I haven’t had to mark a single comment as spam on the page.

About mediatemple, softlayer, anyone ?

Dreamhost, Site Cemetery™

This is some serious hair-on-fire gah my site is down horror scenario. No updates means no sleep for this wicked.

I’m actually quite surprised that they don’t have a fallback server to go to when their primary goes down… The odds of both going down would be quite small…

I mean… I know that shit happens sometimes, but going on 8 hours downtime for me, making my customers get quite frustrated and losing me potential revenue, when it could all be prevented by having a fallback, which, these days, you’d think would be fairly standard…

But I could be wrong.

March 19th, 2013 at 10:55 pmPissed Off Consumer Says:

A really good hosting company I have worked with is http://www.inmotionhosting.com only downside is CPANEL shared hosting. Otherwise speed rocks and you can select your geo-region even for shared hosting. They have really good uptime.

DreamLOLZ

No update in 1hr 15min? My sites are still down…

@Mike250: Dinner break :)

I have a site now up, but no DB access from it.

They’re coming up on a round 8 hours in the next few minutes, for those keeping score at home :)

@Mike250, I’m working on and update now, things should be coming up pretty quickly now. Can you let me know your site? I’ll be happy to check it for you.

I think they’ll need a Geneva Convention when this is all over.

Después de 8 horas un sitio jala y dos no, lo curioso es que el que funciona y otro están en el mismo vps.

March 19th, 2013 at 11:01 pmJeremy Wright Says:

http://dreamhost.com is NOT down. That means that Dreamhost is smart enough to know their platform is garbage and should not be counted on

Totally unacceptable service – and to receive several copies of the exact same message over the past 5 hrs. (all starting only after I had opened a ticket – for a total thus far of 8 hrs. down) is just very poor response & service. I have never experienced so much downtime with any other host over the past 20 yrs. and certainly not anything ever so recurrent or for as many consecutive hrs. of being offline. This is killing me, as we our now in our “busy” season and our company is basically “dead in the water”.

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

http://dreamhost.com/web-hosting/

Regular platforms are for the little people :)

Maybe they used up the last of their AOL Free Internet discs.

Houston, we have a problem :)

mine vps back work right now. well, two questions: When was be next downtime, and the most important: What i will told to my customers ? DH will break 50%-50% with me ?

He said hopefully

Thanks For Host Master at now our websites in live….But Please don’t repeat this type off issues…

This is what happens when you run your server software on an iPhone.

Oh so the outages are sporadic now??? homie-vserver323 is dead as the hamsters powering this joke of a host!

My 20+ sites are back up.
12 hours downtime = 12 days of free hosting… great.

Ok my site is now back up. Very Curious that I finally submitted a help ticket and less than 4 minutes later my site comes back up. Squeaky wheel gets the grease? I suggest you all hit them with your ticket

My site is responding, but the MySQL servers are still slow as balls.

#RevertToDynamicHTMLCirca1995

This is really nuts-very frustrating!

Come now!
My server goes down daily, unable to access any of my sites, services have been down for hours and no reply rfom support. Clients are complaining and unable to contact support on Live Chat. Horrible, now I need to investigate how to transfer all site to reliable hosting provider. Losing Money, every-time this happens I compensate my clients with free hosting, not happy and clients frustrated. Thanks!

@Mark:
“My 20+ sites are back up.
12 hours downtime = 12 days of free hosting… great.”

Don’t count on it… they won’t award any amount over 10% of your monthly hosting bill. In other words… nothing.

Guys, really – of course DH’s site is not down. They set up their business-critical systems with redundancies and fail-overs. You should too. If your business requires 100% performance, YOU need to make certain you have it. If you accept a single point of failure, you accept that you can go down when that point fails. When you set up a simple webhosting account and take no measures to handle the outage when it goes down, you should know you’re accepting that risk.

Hell, _I_ have sites down, too, but I knew going in that if DH went away, or out of business, or whatever, my stuff would be offline. My clients who require uptime have multiple hosting and multiple data centers – I set them up. It’s what I do. It costs them a fair coin, but that is what they require. Don’t depend on a single bargain hosting company to provide that for free with your shared host account.

@Gordo – I suppose they’re vague. They say my next hosting bill – and I’m on a 3 year plan. Who knows. 12 days out of a 3-year plan is inconsequential anyways.

March 19th, 2013 at 11:13 pmcrissdetabarnak Says:

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!

fuck you dreamhost!

Yeah Johan, that’s how I live my life. I bought two cars in case one dies, and two houses in case one falls down. I have two cats, and each cat has nine lives. When I say things, I say them twice just in case one of them disappears.

Yeah Johan, that’s how I live my life. I bought two cars in case one dies, and two houses in case one falls down. I have two cats, and each cat has nine lives. When I say things, I say them twice just in case one of them disappears.

lowered expectations…
lowered expectations…

Only 1/2 my sites are up….8+ hours and counting

Site’s back up. MYSQL is down.

Back online……….. finally. T.T

Now my started coming up one by one…

Still our websites DOWN

See, they made it a nice even eight hours, how thoughtful of them :)

My dedicated server(ds5665) is still down… RTO for pings…

Not online. Business is suffering. Estimated 500$ lost. Thats a lot of income. Migrating the second I am back up. I really should pay for professional service instead of save on crap service.

8 hours downtime, 8 hours in a workday. Coincidence? Probably.

MYSQL still down.

My site’s back online… 8 hours and 25 minutes later…

….and my site is down again! EASY COME EASY GO

It came up as fast as it went down again. For fuck sake, dreamhost wtf.

Only thing responding is this damn fucking status page.

I’m still down.This is getting ridiculous…

Dreamhost, you tease you!

Mine just went down again too. I’m more concerned about my data, all of my websites say error id: “bad_httpd_conf” when I try and go to them, when I log in by SSH all of my data is gone :(

-Jamie M.

Ya se jue el que estaba bueno.
Alguien que conozca un buen psicólogo o psiquiatra

WE should really stop hosting in a developing country like the US where power outages cause large webhosts to go down for 9 HOURS!!!!

still down!!!

again going on all websites downside….

Finally, Both my websites are up and running. I hope this doesn’t happen again. :-)

site went down again. what is this, prom night?

Dreamhost is a proud graduate of the Alfred E. Neuman What Me Worry School of Webhosting

I guess I mistyped anyways, it was 8 hours, not 12 hours downtime.

But then I mistyped when I said everything was back up… everything down again.

I wish I could say Im up and running again. Ouch its been a long time. Ill definitely be looking into some redundancy of my own.

Al Gore Invented the Internet.

Dreamhost is currently working on Uninventing it.

Up and Down Again: A Server’s Tale by Dreamhost Baggins.

LOL!!! Look at my dreamhost status page! Going downhill…. lol! http://tafb.xxx/dreamhost_fuckup.png

-Jamie M.

Is this the worst downtime in DreamHost history? Anyone have a good summary of major downtime in last ~2 years?

@Jamie: BRUTAL LOLZ

My website is on for 15 secs

15 secs? wow, how many customer transactions did you complete in 15 secs?

March 19th, 2013 at 11:34 pmDreamhostSucks Says:

Anyone else considering dropping dreamhost like a bad habit? I think this was the final nail in the coffin for mel

I hope my data isn’t lost. I think it’s pretty poor that a customer would be expected to keep backups of their own data (if you’re going to do that, why not just host the whole thing yourself?).

Yeah, so I have been sitting here thinking…

I’ve liked DH for a long time. However, the day has come…

We are so leaving! and we’re taking the 98 domains we currently have at DH with us ;)

… we plan to keep a small package at AcornHost, ’cause we absolutely love them!

if you just need a solid, small host, I really recommend them. Women-Owned too.

as for our larger hosting needs…

I have decided to abandon hosts in the class of DH. I will keep the packages I have, and maybe get new ones, but just for a diversity of address space, etc.

The new solution?

I am heading to Gandi.net and will experiment with their IaaS offering. ServInt/LiquidWeb have also come highly recommended to me, but Gandi seems more my speed ;~) I’m just way into the whole social values thing…

Best of luck,

Max

<3 We Are Our Choices

@HK Actually, you buy insurance for in case your house falls down, and for if your car crashes. It’s a trade-off between no protection and redundancy.

Now people want to kill me and I’m a fugitive

I’m up. I’m down. I’m up. I’m down. I’m like a pensioner in a Viagra drought. FFS guys, can you please get it up and keep it up!!!!???

My websites have been down for over 6 hours now. Keep up the good work Dreamhost! You’re a rockstar!………… NOT.

@Doug: “can you please get it up and keep it up!!!!???”

That’s what *she* said!

if my house falls down my insurance doesn’t automatically have a new house waiting for me. I still have to find somewhere else to live.

lol, 8 hours. What a farce.

Finally back up after being down all day… sat there looking at:

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.

When suddenly

Reply from 67.205.24.38: bytes=32 time=1102ms TTL=49
Reply from 67.205.24.38: bytes=32 time=865ms TTL=49

As to all the people complaining, I’ve been with Dreamhost for over a year now and have never had noticeable/significant downtime. If the power went out and both the ‘network devices’ which I am assuming are routers or switches were killed, it’s not their fault. If you generate money from your website then you need to fork out a bit of that and get something that is properly redundant.

Thankyou (Even though DH probably don’t read this) for the constant updates on the issue and providing an easy to find area to see what’s going on… better then a lot of places where I have been left in the dark. The downtime is inconvenient but what’s important is that it was resolved and I knew what was going on.

I’m getting the hell off dreamhost as soon as mysql is back up, already signed up with media temple and am up and running.

Kai Waugh Says:
Reply from 67.205.24.38: bytes=32 time=1102ms TTL=49
Reply from 67.205.24.38: bytes=32 time=865ms TTL=49

LOL! Like one second of latency will be acceptable to your customers ;)

-Jamie M.

Ok, this is a joke. I understand shit happens and something goes down. But this long? Seriously? I never had this issue with my last hosting company. Guess I’ll be switching back in the morning.

Dear Dreamhost,

This is the last straw. Will be migrating everything to EC2 as soon as I can download a backup via FTP.

Kind regards,

DownDownDown

Nothing a dose of e-Viagra can’t fix.

“I’ve been with Dreamhost for over a year now and have never had noticeable/significant downtime”

You were just down all day, I’m not sure how you can make this statement.

Any reputable hosting company would have systems in place to prevent an outage of this magnitude and length. As I posted earlier, I had this happen once before with DreamHost and we were down for nearly a week. That’s completely unacceptable. An outage of an hour or less once in a blue moon is understandable – outages of half a day or more? There is something rotten under the hood here.

Notice how they said – several hours after this started – ‘thankfully no data was lost’ – for many hours they thought there was a real possibility our data was gone! That means that is a risk in the future!

Fuck that noise. AWS here I come.

Nice work! Thank you.

i’m just gonna take the opportunity to plug gandi again (https://www.gandi.net/hosting/vps)

and while we’re at this, we might think about shifting our sort of business more intentionally.

dollars = leverage, yeah?

so, we should totally be demanding responsible/ethical business/operations

and ps. next time you go to GoDaddy, just don’t do it!

@KC This is probably the longest downtime I’ve had in my four years with them. It sounds like everything that could go wrong did go wrong. Still, there’s nothing I can do but wait, and hope my server didn’t lose any data or anything. :/

“Our 100% uptime guarantee means we take uptime seriously.”

I call bullshit.

So for those who are not crying right now… What are the proper options for redundancy and what would it run? Would I be doubling my hosting costs or is there a service that can pop on when needed and not cost as much as another plan with another hosting company.

Looks like one of my MySQL servers is up, the rest are still down.

Hey Dreamhost, of the other companies using your colo, how many of them are still down? Very few, right?

You can blame the initial outage on the power going out and even excuse away a little more because you bought cheap routers – but this extended delay is entirely your fault, and you know it. No excuses can be made except for the fact that you’re running on ancient, rotting technology.

This is ridiculous!! Down for 10 hours or so now. Un***** believable. We^re out of here

We are glad to say we got it figured out. We didn’t pay the electric bill last month. We are mailing in the payment in the morning so your services should be back online in about 7 business days when the check clears. Thanks for your patience.

Dreamhost

Ok, my website is being served now. Ping response time is 2144 ms !!!
Sheesh.

again, cause i’m still feeling pissed:

(https://www.gandi.net/hosting/vps)

starting at $16/m for 1 cpu core, 512 mb ram, 12gb on raid 10, ~3tb/m @ 10mb/s

… and get this, they refund for downtime

Jamie Says:
Kai Waugh Says:
Reply from 67.205.24.38: bytes=32 time=1102ms TTL=49
Reply from 67.205.24.38: bytes=32 time=865ms TTL=49
LOL! Like one second of latency will be acceptable to your customers
-Jamie M.

That’s because I’ve got satellite Internet. The ping is at least 800 to pretty much anywhere.

DownDownDown Says:
“I’ve been with Dreamhost for over a year now and have never had noticeable/significant downtime”
You were just down all day, I’m not sure how you can make this statement.
Any reputable hosting company would have systems in place to prevent an outage of this magnitude and length. As I posted earlier, I had this happen once before with DreamHost and we were down for nearly a week. That’s completely unacceptable. An outage of an hour or less once in a blue moon is understandable – outages of half a day or more? There is something rotten under the hood here.
Notice how they said – several hours after this started – ‘thankfully no data was lost’ – for many hours they thought there was a real possibility our data was gone! That means that is a risk in the future!
Fuck that noise. AWS here I come.

If you could use a very small amount of common sense, I was obviously referring to any time prior to now as we all know that I have experienced downtime today, otherwise I wouldn’t be here and that comment would seem stupid.

I suppose the outage was for a VERY long time but it doesn’t bother me too much. If it happened again however I would be thinking twice.
As to data loss, only an idiot wouldn’t keep backups of all their important data. I keep a MySQL backup weekly and all the data from my site stored on one of my External HDD’s just in case anything was to happen. If you can’t be bothered backing up it’s pretty easy to automate the process, too.

Site up, DB down

Please address http://pokemonshowdown.com
we have all been staring at the screen for hours. I know you guys are very busy and problems like this are out of your hands and completely random. But if you guys would do something about that site a bit faster, we will all be very grateful!

Champion DesTroy Says:
Please address http://pokemonshowdown.com
we have all been staring at the screen for hours. I know you guys are very busy and problems like this are out of your hands and completely random. But if you guys would do something about that site a bit faster, we will all be very grateful!

That’s up for me. Issue might be your end… Clear cache and/or restart browser.

Kai Waugh: My apologies. I was getting similar ping times (1000 to 2000 ms) to my dreamhost server (and my ping times are usually under 100ms to most places), so I figured that was the case with you too :)

-Jamie M.

@Dreamhost: DON’T BE F—IN’ WITH POKEMON. THAT’S JUST PURE EVIL

Yes, Dreamhost, please, forget the rest of us and just get pokemonshowdown.com back online. Because the world will end if we can’t play Pokemon in the middle of the night.

March 20th, 2013 at 12:05 amFOUCK YOU DREAMHOST Says:

FOUCK YOU DREAMHOST

Looks like others are popping up… *cough cough* pokemon *cough cough* but still no luck here. I expect soon. Thanks for keeping on it guys and fingers crossed that you guys learn how to prevent this type of outage again. Seems a bit frequent lately…

March 20th, 2013 at 12:06 amFucked by Dreamhost Says:

EVERYONE REMEMBER TO DEMAND A REFUND BASED ON THE
“100% UPTIME GUARANTEE”

Dreamhost needs to take a big hit in the pocketbook if they’re ever going
to learn that they need to invest in reliable hardware, redundant systems,
and competent employees.

It’s clear at this point that they have none of this.

Site is up now but DB is down. Pfft.

Yay, site is back. Finally. That sucked.

Jamie Says:
Kai Waugh: My apologies. I was getting similar ping times (1000 to 2000 ms) to my dreamhost server (and my ping times are usually under 100ms to most places), so I figured that was the case with you too
-Jamie M.

Ah ok. My site seems to be quite responsive, little or no noticeable difference from usual (Although with this Internet everything is pretty rubbish :P)

not up yet, this sucks

More of my sites are back. Another MySQL server back up.

Very worst. Still not up yet. we lose our biz.

For the record, my data is back! Looks like they rebooted the VPS and everything is back to normal, data is there, programs, settings, websites :)

Hopefully it stays up now!

-Jamie M.

No sites backup here yet. Disgusting

I host multiple sites on my VPS. We have been down for over 9 hours now. My clients are starting to get extremely frustrated, and I’m wondering when we’ll see resolution to this. My VPS still seems to be down. Does anyone know who I might connect with to solve my issues? Multiple reboots and still nothing….

Mine dedicated server(ds5665) is still down…Its nice to hear that some of you are back online…

None of my sites are up. After all of the false hope from each update I’m still sitting in front of a monitor, like a fool. It’s 3:17AM, and I have to be here until the sites come back up. How awesome is that? I don’t like sleep anyway- it’s overrated. Going on 23 hours uptime, much better than DH.

8 hours later, and down again.

Good luck folks. Hope it gets fixed soon.

HEY GUYS, I MANUALLY REBOOTED EVERYTHING IN THE CONTROL PANEL AND EVERYTHING CAME BACK UP.

Perhaps we’ll smush your name a little, and mix it up a bit…. how about D’OHost?

And I’m up! Hopefully it stays that way :)

Still down. Demanding a refund. I’m sure everybody is doing their best, but that’s not good infrastructure. Worrying.

IPv6 connectivity??

I’m so glad a lot of you guys are up. I hope I get up soon. I gotta get up to get down (to sleep).

Actually 10 hours later of downtime now. Getting nervous.

my site is back up

yay! my db is back! :)

Im still limp. :-(

my site is still down. :(

Almost 12 hours later and one out of 4 hosts reachable.

It’s all too much, might as well try and sleep it off.

Thanks for the nightmares, ScreamHost.

I am creating a ticket demanding a discount on my next bill (I pay monthly). Using the Panel resource status screen I captured and measured my downtime: http://tafb.xxx/dreamhost_downtime.png

Wish me luck :)

-Jamie M.

oh gosh is this still going on?!

Am back online….. am so glad……. worst timing ever…….only down from 8m -5pm in Australia.. lost the whole business day..

Still down. Still f** up. I won’t take this s+`anymore. Will move ASAP to a provider that deserves to be called “provider”.

Hey guys reboot your server via the control panel and your sites might fix.

Try the reboot, still not working. :-/

We didn’t have to reboot anything for our two VPS’s to come back up… rebooting from the panel probably just makes you feel busy until they get around to getting the machine you are on working. You’d be better off making yourself a sandwich.

I’m back online.

It’s nearly 4am eastern…. all sites still down.

So we’re not being billed for this downtime rapidly approaching 12 hours right?

Psst. Marathon fault tolerance.

My stuff’s back up, after only just over 9 hours of obliteration. It’s been fun, but I’m out!

chii: From Dreamhosts 100% uptime guarantee: ” … is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance … we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee.”

-Jamie M.

March 20th, 2013 at 1:11 amone of DH user Says:

From Dreamhosts 100% DOWNTIME guarantee…!

I’m setting the over/under on all sites being full restored at 72 hours.

Anybody still believing their status updates that imply this is only minutes away from being fixed, isn’t very smart.

At this point I think support would be courteous enough to provide a status update at least every 30 minutes.

March 20th, 2013 at 1:16 amone of DH user Says:

DownDownDown,

That’s what happened 7years ago at Floria data center!

Back up. Good luck everybody.

I restarted the VPS and MySQL DB about half an hour ago. Not sure if that helped but it didn’t hurt any so may as well try it.

10 hours later … Finally works … I hope this is compensated by Dreamhost…Thank you…

My site is still not working. Can you solve it? thanks

Still not up. My developer is in Germany. We are in the final days pre-launch and this is setting us back on our launch date. Since 3pm PST with this! Can I pay for redundancy, please? Because this is costing me hair which of I don’t have a lot left!

Not yet. Can ping and telnet, but cannot ssh or open web

HEY! And we’re back! Breathing again… Thank you for the fix! (Still would like to be able to pay for a redundancy option, though!)

March 20th, 2013 at 1:35 amArafin Shaon Says:

I’m not getting why its taking too long to fix this issue………..

Still not up yet. Once this issue has been resolved, I will be taking my business elsewhere.

No joy here. still.

Come on.
………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

March 20th, 2013 at 1:38 amArafin Shaon Says:

opssssss the problem has been fixed :)

It is still down here – that is 12 hours – will you fund that thousands we have now lost?

Looks like sites are up, but mysql isnt connecting :-(

my VPS still not working, hosted sites still down in the middle of the campaign :(

Over 12 hours down is not worth staying with DH – just instructed my team to see options for fast move elsewhere – Lets face it, even the expensive hosting sites are cheaper than the losses we have suffered as a result of 12 hours out…

Ours is still not up

My boss will kill me. Who can warm my little heart. :(

It seems the sites are available, but the mySql connection is very slow: posting to the database takes over 30 seconds, and I am the only user at the moment.

Our website still not working! You guys are back online? Why not us?

Online again finally at 0435 Eastern. Thank the Lloyd Jeebus.

Sites are still not up…so frustrating.

Back online, but 608 min downtime because of a power disruption????

My Website comeback, thank you

all my sites are up now.. Thanks

Directories are still not writable.. cant work on ftp

Have been experiencing way too many outages lately an now 10 hours!!! Unacceptable! Shame, DH has a lot going for it but it’s back to MediaTemple we go!!!

12 hours down, and still waiting. /sigh

Just to say something positive after some of these negative posts here: this must have been a good learning experience for the DreamHost team, and I bet they will do anything to prevent this from happening again or at least keep the down-time as short as possible.
If you still think the grass is greener elsewhere, you might be disappointed again. So, just give them the benefit of the doubt, and see how they handle it after everything is fixed again.

my databases are unstable! : (

ALL THE CRAZIEST-WEIRDEST THINGS HAPPENS IN IRVINE.
YOU GUYS SHOULD HAVE A FULL CHECK OF THIS DATACENTER.

Myself and others hosted in Irvine are down AGAIN! Yet this is marked as resolved. FFS…

All down again

I was up for like… 15 minutes :/ Now everything is down again.

This is ridiculous, this is the fourth month since I have signed up with Dreamhost I have had to credit clients more than $300 due to downtime and it is always the Irvine data center. I am shopping for a new provider.

Down again. Very frustrating. At what point will partial refunds be given?

Down again fot 15 minutes – no up while writing

It’s been 14 hours since my site’s been down. What’s going on Dreamhost?

down and down -.-

MY website are STILL down?!

Jesus effing Christ. Fix your god damn servers. You’re causing all of us to lose money. As soon as my site is back up, I’m leaving Dreamhost. You guys seriously suck.

Dreamhost has never given me a credit or a refund.

Just received this reply from Dreamhost when I told them all sites were down again: “This email is to let you know that we have received your message and will be replying to it within 24 hours.”

I should send THAT to my clients next time and see how long I can stay in business that way.

Ok..So party started again… Just had a relief for 45minutes.. Now again!!!! Really don’t know how to answer my clients…

Come August I’m done.

Our websites are still down. So far its been over 14 hours and your website indicates that the Irvine location has been resolved. Please have this problem escalated to someone who can get all this resolved or be honest with us and tell us we are screwed.

WTF? Another outage now!? Just to add insult to injury and piss us off more…shit my site was down for 6 hours and 37 minutes today! Now this.

ARE YOU SERIOUS?! ANOTHER OUTAGE?! god

FYI through Scoutapp I can see that my PS server is running. So it must be their networking equipment again not letting traffic through.

My site is down again… This is nightmare….

March 20th, 2013 at 4:46 amcrissdetabarnak Says:

WHAT THE FUCK!!!!!!!!!!! you fucking*?%$#&?%$@#?*$&(%*?)(O?&%?*U%Y#$@ StiLLL DOWN!!! AAAAAAHHHHHRGGGGGGGHAHHAAAAAAA

We are down again!

uh oh, here we go again

Maaa ki Chooot tumhari. Madar chod.

According to PingDom my downtime has been 10h 2m

My site is down

WTF? Almost a day now.

Down again

Perhaps we need to relocate from Irvine. It’s beginning to give me convulsions every time I hear that name. :p

Hope you guys get to the bottom of whatever’s happening there.

Right before this all went down again I sent a mass email to all my clients telling them we’re all back online now. I guess I’ll have to embarrass myself a little more and send a follow-up email. What do I have to lose? Oh wait, just my income and credibility.

Down again?? This is crazy. My clients are going to leave me.

you can get REFUND for all UNPLANNED downtime! this downtime is unplanned, so contact support and ask for refund for this time. i did it last time, they did credited me ;)

even if amount is low, for example $1, but if they will process 1000-5000 refunds a day THEY WILL TAKE SOME MORE ACTIONS TO PREVENT THIS FROM HAPPENING

we must flood them with refund requests, even for small amounts!

March 20th, 2013 at 4:51 amAngry Webmaster Says:

This is an absolute disgrace! I am losing money by the minute because some fools can’t keep a hosting service going.

Using Dreamhost was the stupidest decision I made in 2012!!!!!!!!!!!!!!!!!!!

We are under attack!!

Seems to me that dreamhost ought to have a proper backup plan for events such as this! 1 hour of downtime is more than enough. But an entire day?? This is absurd!

Okay, so I spent all night telling clients they are going to be back up and they were for about 2 minutes. As a new day starts, I don’t want to spend the whole day putting out fires. This is not resolved. All sites and mail are still down. Unacceptable….

Connection to 67.205.11.120 Failed

site was up; now back down; when working earlier pages loading very slowing and getting critical errors related to loading time; just got a vps for its amp capability; very frustrated

Guys, I’ve sent you multiple friends to sign up hosting plans with you, I was VERY happy with you as my main hoster in comparison to jerks like bluehost, nevertheless I understand an outage is unpredictable but redundancy is there for that

Could you please seriously FIX THIS as soon as possible and/or give credit (money back) for that frustration

It would be a great sign of humanism, compassion and professionalism

thanks guys I hope to hear from you

look the stats don’t lie http://status.alienatus.com

Just saw this post on Twitter: “My @linode server has been up for 1186 days. I guess this will be as good a reason as any to reboot.”

I guess you get what you pay for.

Like I said before, getting hosting from Dreamhost is like buying a guitar at Walmart. The best one they’ve got is still a piece of crap. So is my $249/mo Dreamhost PS.

Is it down AGAIN ?

Maddaiiiiiiiiiiiiiii!!!!! Cazzo!!!!!! Di nuovo nooooooooooooooo!!!! ()/(&/%$£%”$&/)(=(/&%$&£$%/&(/)=)(&/%$%£$&(/)(=?)=(/&%$&%$&£%$&%/(&/)=?)=)(&/%$$%&/()=?)

Well luckily I host my companies actual website else where since DownHost is too dumb to have redundancy. The bad news with that is my support ticket system is working and I have received over 200 support tickets blaming us for the problem. I run a service on my DownHost server that should be accessible by my clients 24/7 and when it is down and I hit a certain downtime as reported by PingDom which I am .02% from reaching I will have to credit the clients ALL ANOTHER MONTH due to DownHost ignorance. My wife is an attorney who has argued cases before the US Supreme Court and is up and looking over my Terms of Use I agreed to in order to see if I am due any legal relief.

All my sites are down!!!!!

I can’t believe

How time we need to wait…. please!!!

Aaaaand we’re back down.

Man downtime for more than 5 hours is not acceptable…and again all the sites are down…i have hosting with multiple companies but have never faced so many issues with any other company.

98% uptime

Esto es demaciado !!! Creo que cambiare de Hosting

Please give us an ETA..so we can set our client expectations.

WAS UP FOR 4 HOURS — NOW DOWN AGAIN …. COME ON GUYS

Again????????

I am fed up with this!!!!!!! wth??????

for me on brazil its 2 days in the roll.. omg DH.. i just moved in.. why u do this! my clients are pissed!

March 20th, 2013 at 5:03 amcrissdetabarnak Says:

Please Chris keep us posted on your wife’s investigation.

NOssa até agora todos os sites fora do ar!!!!

Unfortunately the Terms of Use are pretty locked down, only recourse is if you can prove negligence on the part of DH which has caused harm to your business or your business’s reputation.

wait… what? another power disruption? dudes…

I’ve got the power

Any news?

I’ve never had so many problems with a host before. We experience constant lag and SQL errors throughout the day.

Now, after a second day of total downtime (something I’ve literally never seen before with any webhost in over a decade), I can definitely say that I am leaving this place and finding another host. This is completely absurd.

13 hours offline

“We sincerely apologize for the inconvenience.” ??????

Hoy es mi día de pago, dreamhots me ha mandado el correo de recordatorio de que tengo que pagar…

son unos sinverguenzas!

So people? Here we go AGAIN?

WTF???????????

Again All Sites are Down Side…..Please resolve all issues ASAP.

Down last night and down again during one of the busiest times of the year for us. Not acceptable!

Kinda funny, I have a back up for my PC that will run it for 6 hours before shutdown and my cable modem from Comcast now even has a battery in it so if the power goes out the cable modem will stay active for 8 hours. They make these things called emergency generators that real data providing companies use, don’t think the Walmart you are getting your equipment from carries them though DownHost. The other hosting company I use has its servers based in New Jersey and uses back up power as well as Satellite backbone back up and didn’t have one moment of downtime during Hurricane Sandy. They must be doing something right. I am looking to see if they have VPS and Dedicated server services.

I would like to see what happened if dreamhost.com were down for this long. Something tells me the outage would have been fixed much faster…

My site is down from past several hrs.
No one is replying to support tickets even.

Dreamhost didn’t even had courtesy to send an email stating this problem..I have to figure out everything myself.

What kind of support and company is this?

@dhstatus More sites coming back online: http://bit.ly/10eRyNE Thank you all for your continued patience!

Bullshit!!

stunning incompetence here

I think this may be some sort of sabotage…

My website was down for 10 hours yesterday and it remains today as well. My website is not up since your Yesterday’s maintenance. It really panics.

Please someone tell me, is all the data of my sites will be ok?

My website is down yet.

Come on, work hard guys

And you are asking me to pay for another year this week? I understand there is an issue but more than 10 hrs? Hope there is a compensation for this kind of downtime.

Wonderful. Power has been restored but my sites still don’t work. Exactly what I needed.

You have got be kidding me. Two power outages? Who is running your datacenter? a bunch of fucking monkeys?

14 hours and counting

Power has been restored but my sites still don’t work

You might think it’s fixed but my site is still down…..

Half hour to switch over to the backup generators? Shouldn’t they have kicked in automatically? Please explain.

is dreamhost really in korea or something? is that the current excuse?

Odd thing is that Edison International (Power company for Irvine, Ca) is reporting no outages or problems within the past 24 hours.

omg…. two time a day??????????????//

LinkedIn is down too…..can you believe it, they must be hosted at DH.

Oh no, not again.

……………………………………..________
………………………………,.-’”……………….“~.,
………………………..,.-”……………………………..”-.,
…………………….,/………………………………………..”:,
…………………,?………………………………………………,
………………./…………………………………………………..,}
……………../………………………………………………,:`^`..}
……………/……………………………………………,:”………/
…………..?…..__…………………………………..:`………../
…………./__.(…..”~-,_…………………………,:`………./
………../(_….”~,_……..”~,_………………..,:`…….._/
……….{.._$;_……”=,_…….”-,_…….,.-~-,},.~”;/….}
………..((…..*~_…….”=-._……”;,,./`…./”…………../
…,,,___.`~,……”~.,………………..`…..}…………../
…………(….`=-,,…….`……………………(……;_,,-”
…………/.`~,……`-………………………………./
………….`~.*-,……………………………….|,./…..,__
,,_……….}.>-._……………………………..|…………..`=~-,
…..`=~-,__……`,……………………………
……………….`=~-,,.,………………………….
…………………………..`:,,………………………`…………..__
……………………………….`=-,……………….,%`>–==“
…………………………………._……….._,-%…….`
……………………………..,

Wow. This is stunning. I just sent people back to my site and now it is gone again. You are putting people out of business! I only hope my clients are loyal enough to weather this with me. :(

What happened again. It was working 2 hrs ago. This is UNACCEPTABLE. Even during Sandy @ East Coast, my local servers were down only 8 hrs. Why a Hosting Company can not have Backup UPS systems, Generators? I will take my business away from Dreamhost the minute I download my sites (I have 7 sites on Dreamhost servers).

LinkedIn just came back up, but they were down for the last 15 minutes too.

FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.
FOR THE ONES IN VPS REBOOT FROM YOUR CONTROL PANEL AND YOUR SITE SHOULD BECOME ONLINE. I’VE DONE SO AND WORKED.

what to do?

————————————————————————
– After reading this response, please consider visiting
– the survey below to comment on its quality. Thanks!
http://www.dreamhost.com/survey.cgi?n=LOL&m=LOL

– If the service you received from us was exceptional, please consider
– tweeting your love for @dreamhost. It’ll warm our hearts, soothe
– our souls, and get you good karma at some point down the road.
————————————————————————

LOOOOOLLL

We have several sites running on dedicated and shared hosting, we are paying hundreds of $$ every month but my ass is full of …….. What the hell, who will pay all these loses. I agreed that a bunch of F**K monkeys are sitting in data center and playing cards.

BKP database to need!

Please Hurry up to solve it, two days of “down” is not acceptable

I am leaving DreamHost after this

March 20th, 2013 at 5:24 amRudy Simpkins Says:

Back down again. WTF.

i can not open my site? what is going on man?

I am out of dreamhost by the weekend….

hostgator is the best IMO as many others. They even have 24×7 support by PHONE, CHAT… I will host for you for $5 a month haha unlimited everything.

Really again. WTF.

Downtime is why I left 1and1. I really don’t want to move the sites again…….

Dreamhost said: “We’ve had one rack of machines giving us a particularly tough time.”

Nice rack!

Just wanted to second that rebooting VPS from the control panel worked and I’m finally back up and running. If you’re on VPS, it’s definitely worth a shot.

Please bring up your Dedicated boxes first..as we are paying the most money for your hosting services.

Update March 20th 5:15am: We are in a much better status
Update March 20th 5:00am: Power has been restored
Update March 20th 2:16am: Most sites and services should be back up now,
Update March 19th 8:30pm: We’re having some success

“WE ARE NOT STUPID”

@QUIERO CHAT – Thanks for the tip. Rebooted my VPS and now I’m back up. This is beyond ridiculous though. I want more compensation than 1 day for each hour of downtime. Being down certainly cost me more than that!

Our website still doesn`t work :(

Thanks to “QUIERO CHAT”…

THE SOLUTION IS:
1) If you are on VPS (Virtual Private Server): REBOOT IT and it will become online.
2) If you are on PS (Private Server): REBOOT IT and it will become online.

THAT SOLVED MY PROBLEM!

I think they need to bring ALL services UP, but if money talks, they need to bring DEDICATED SERVERS first. GUYS BRING US ONLINE PLEASE.
Look at the messages, you lose power, bring power back but routing tables are disrupted…. Hey the command is WR MEM when you make a change on the router. When power comes back, it ready WHAT YOU SAVED.

My VPS were back up without rebooting. So my systems run again.

Yes..Reboot brought back my dedicated server online now… Have to see how long it will stay!

How can they claim 100% uptime for their dedicated servers? They have a 100 percent uptime guarantee. This is false advertising. A class action is in order.

And my client is finally backup, just in time to make a backup so we can get them switched to another hosting company.

Trying to reboot as suggested and guess what, Not rebooting! So over this!

Meus sites estão voltando pro ar aos poucos… \o/

After becaming online couple of hours ago, it is offline again.

I have restarted it for the first time in months and it is doesn’t work.

Please, I can’t deal with this anymore.

We’re 12 hours offline.

I was up briefly, but now back down for 1+ hr. C’mon guys.

Back up for now….

“Update March 20th 5:15am: …it is a bit of a process.”

HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA…..

எல்லாரும் சேர்ந்து கும்மி அடிங்கடா… என்னத்த சர்வர் வச்சிருக்கீங்க…. ஊத்தி மூடிட்டு போங்கடா…

Not sure why, but logging into my account via ftp and opening all of my sites folders one by one seemed to make them come back online. Whatever, I’m leaving dreamhost this weekend.

Not sure why, but logging into my account via ftp and opening all of my sites folders one by one seemed to make them come back online. Whatever, I’m leaving dreamhost this weekend.

Not sure why, but logging into my account via ftp and opening all of my sites folders one by one seemed to make them come back online. Whatever, I’m leaving dreamhost this weekend.

Not sure why, but logging into my account via ftp and opening all of my sites folders one by one seemed to make them come back online. Whatever, I’m leaving dreamhost this weekend.

Not sure why, but logging into my account via ftp and opening all of my sites folders one by one seemed to make them come back online. Whatever, I’m leaving dreamhost this weekend.

Back up now.

Thanks

Now let us keep it that way!!!!!!!

Hi

Mysql Is also note working.. they said.. onle ftp had problems..

what… everything online? this is not possible on Dream Host…. Check twice. i think this is DREAM

My website hosted on a shared account is now online! Tnks for feedback

We are the world
We are the websites
We are the sites that Dreamhost fucked
And now we’re livid
There’s a choice we’re making
We’re saving our own sites
It’s true we’ll make a better day
Without Dreamhost

Tried to reboot MySQL VPS – Failed!!!!!!!!!!

It is time to start backup, download and run to another webhost before it goes down again…

Backup now!!!! Aproveitar que voltou tirar bkp nesse instante.. \o/

Dreamhost. Your inability to be prepared for something as common as a power outage has not just shut down our site. It has shut down our entire business. We will be expecting restitution for this.

This is a comedy of errors. How about hiring some adults to run your company …

Site/forum on a shared server now appears to be back up – until the next power cut.

Only 60% the my sites UP. 40% down! :-/

The site has the same problem again last night. It worked for a few hours and now no longer works. How long will it take to resolve this situation?

How can you laugh when you know I’m down?
How can you laugh when you know I’m down?

It was super easy to set up a site at BlueHost.com

“Dedicated machines are next on our list to get back up.”

This means that when you move to a Dedicated you have less priority when maintenance is needed?

Seeams a really “Great Policy”.

Please take closer look at my DS.

! Current Outage: DreamHost web server host ‘sadalmelik’ being moved to new hardware (updated 3 hours 3 mins ago)

Hello,

You are receiving this message because your web hosting services are being affected by hardware maintenance due to issues with your shared web hosting server sadalmelik.

Due to issues with the RAID controller, we are replacing the hardware for this server and will be restoring all data from our backup server to the new hardware. Currently, we are working on connecting the backup server, and in the meantime you may be experiencing downtime with your websites or may be unable to access your server. We apologize for the inconvenience, and we’ll update this post in the next hour or as soon as progress is made, whichever is sooner.

A little bit about the restore process: once the backup connection is established, we will reload your web service and will serve your sites again using the data on the backup server. The data will be mounted over the network to allow your hosting services to remain online for the duration of the process.

Thanks for working so hard to get this resolved – sounds like a total nightmare all around. My sites all seem to be back up now.

Once the dust has settled, though, I think you’d be well advised to start looking around for a new, more reliable data centre. I know what happened there wasn’t anything under your control, but particularly with two power disruptions in one night, I wouldn’t be too confident about continuing to use that one in the future.

Also, I would love it if you could find a way to build in a bit more redundancy in your networks, so that even if one data centre were to fall into the sea or something, the sites it hosts could be mirrored in a different one and would automatically switch over in the event that the primary one failed. Even if it was an add-on option available at an extra cost. I’d pay for that…

Hey Everyone,

I worked in IT now for 10+ years (not for Dreamhost) but every once in a great while the stars align for a significant outage.

No one wants everything restored more than DH, and they’re the ones that haven’t gone without sleep in 36 hours trying to make sure everything’s coming back up.

While the derogatory comments might make you feel better they’re not going to speed anything up so let’s trust that they’re working as hard as they can on this particular issue and remember that 99.999% up time still accounts for 8.76 down hours a year.

Nice, my sites are down for 12h

lol at all the mad nerds

no one will miss your shitty furry slashfic sites

my websites (VPS) seems to be up and running again.

seriously going to move to another service – DH is surprisingly shoddy.

@Jim –

What are you talking about? 99.999% uptime equates to 5.2 MINUTES of downtime per year. This single outage has already taken them down to 99.8% uptime.

@Jim
+1 for you, and ditto on the +10 years.

@Mike: Barbie says math class is tough. :)

Yeah I agree with what someone else on here said. All shared hosting seems to be back up, yet dedicated hosting still offline. Get what you pay for has been thrown out the window.

My site is back up. Thank you! :)

Good luck Dreamhost. Thank you for working diligently on this.

Our company’s application is still in development so this is a real eye opener. We’ll be designing our web application to be fully distributed and redundant, to handle situations like this.

The tuba MySQL server is still down. So, no database connection.

Down again…wtf. Has come back up and down twice now.

Let’s all remember, no hosting company is perfect, and there are several factors at play here that our out of their control. Those of you talking about redundancy should know by now that YOU should be implementing your own redundancy if your sites are really THAT important.

See you later, whiners. Your exit will free up some resources on my shared server. Bye!

I see Dreamhost personnel are now on here doing pseudonymous damage control. How touching :)

The domains connected to my account that are on pristina (and piranha as databases are concerned) are still down. Any news

normally I spend all morning at work dicking around and posting stupid shit on my website with “friends” that I’ve never met. Today I’ve spent all morning refreshing this page and checking to see if my site is up. The end result is the same, but what I’m doing today is far less fun.

! Current Outage: DreamHost web server host ‘sadalmelik’ being moved to new hardware (updated 3 hours 49 mins ago)

I’d like to organize a class action lawsuit… who’s in??

Still down. Glad to know I’m in the 5% of VPS that aren’t up yet.

The action which will speak the loudest is the action of cancelling your account and moving to a different hosting provider.
Anything else is just whining.
I am moving…I’ve given DH more than enough chances to retain my business, and this is the last straw.

March 20th, 2013 at 7:39 amMatt Van Andel Says:

For anyone looking for Dreamhost alternatives, here a few good options to consider:

> Rackspace Cloud Hosting
> MediaTemple – mediatemple.net
> A Small Orange – asmallorange.com

I want to point out a few things, though. Dreamhost has dreadful reliability and pathetic email-only support, but the features and benefits they offer WILL NOT be matched by anyone else, period. You will need to balance your desire for features and benefits (like unlimited storage and bandwidth) with your need for stability and guaranteed uptime.

If you have the money, Rackspace is the way to go. They offer a staggering amount of services like failover and CDN caching for those who need to ensure their site NEVER goes down (again, if you have the money). Rackspace is one of the biggest and most trusted hosting providers in the world, second only to Amazon Web Services.
Comparable Pricing: $100+

MediaTemple has a fantastic track record for up-time, and my only complaints with them are that they use Plesk for their control panel and don’t have quite as much transparency as Dreamhost does (a few years ago we found out the hard way that MT was secretly throttling VPS’s CPU during peak hours and bringing our entire ecommerce site down – not even they were aware that was happening). These days they guarantee CPU & memory availability and have resource monitoring tools very much like Dreamhost’s. Oh, and you can get someone on the phone when you need it.
Comparable Pricing: $50+

A Small Orange is the small potatoes here… but they offer failover solutions in the event of outages. That said, they have pricing models very similar to Dreamhost – making them a natural choice if cost is an issue.
Comparable Pricing: $25+

My VPS is still down. I’m glad they’ve restored clients that are not paying $50 a month for the worst VPS service the world over.

March 20th, 2013 at 7:45 amMatt Van Andel Says:

And for the record, the reason I’m leaving Dreamhost is because I *NEED* the reliability.

I have a lot of clients who aren’t bothered by one or two major outages a year, but I also have a lot of clients whose very livelihoods and legal compliance hinges on availability. Neither I, nor they, can afford Dreamhost’s seemingly annual 6+ hour outages. It makes me look bad, it makes them look bad, and it’s utterly unthinkable.

I loved you, Dreamhost. But I. Am. Done.

“All services have been restored with the exception of Dreamhost Objects” – What? no, this is not true. My VPS is down again.

DreamHost may believe this power outage was beyond their control. It was not. They may want us to believe there was nothing they could have done, but there was.

A good hosting provider would have had at least ONE of the following: 1) Battery backup for at least an hour, 2) Generators for backup power that kick on with power loss, 3) Live backups ready to go in another data center. A great hosting provider would have all three. DreamHost had NONE OF THESE.

I will be seeking out a new host that can actually deliver on their promises for uptime. I’ve put up with so many outages over the 10 years that I’ve been on DreamHost, but this latest one is absolutely inexcusable.

Their policy says they will compensate a free day for every hour they’re down (which would be A LOT of free days)…but it also says they won’t compensate more than 10%. 10% of 30 days is 3 days…or 3 hours of downtime. That’s less than half of this outage! So everything beyond 3 hours means we get 100% screwed on that downtime with no compensation.

Screw you, DreamHost. SCREW YOU!

Dear Dreamhost,

This isn’t working out.

Sorry, it’s not me, it’s you. :)

Sayonara!

There must be something wrong with Timestamp and/or Timezone

As a matter of fact, for example, using PHP I can’t set anymore the timezone to ‘Europe/Rome’

Everything worked just fine until the issue on datacenter.
So, I believe there must be something wrong with the system configuration.

Does anybody is experiencing the same issue?

@assitweet: Nothing would surprise me at this point.

Assume nothing. Especially nothing positive.

Anybody else still down? I’ve had FTP for an hour now but still nothing in a browser.

“All services have been restored with the exception of Dreamhost Objects”

False! MY VPS is still down, and I have rebooted it four times. Servers HAVE NOT BEEN RESTORED! Please stop putting out wrong information.

could this really be 17 hours and counting for some people?

According to the latest update “All services have been restored with the exception of Dreamhost Objects”

I don’t use Dreamhost Obejects and I’m still down.

Disclaimer 2.0: Comments posted here may be ridiculed behind your backs by DreamHost staff. Please note that these posts cut no ice with them at all, lozerz.

I’ve been down for 17 hours now.

@GD That would be me. Going on 18th hour now.

Our site is still down. We’re a non-profit so Rackspace is a bit out of our budget but this is bad and we may have to consider other options. This outage is inconvenient but if this was a launch day for a fundraising campaign it would be a disaster.

Well, this is beyond frustrating. All of our sites are still down. Do you think Dreamhost crew has packed up and left the country? Anyone know if they have insurance for this kind of thing to cover our financial losses?

@G D

ok, now I’m sure it is because of the the issue on datacenter.
I have two hosting plan with DH and they are in two different datacenter.

In Italy it’s 4:03 PM

Not Working => http://www.assitweet.com/test_timezone.php
Working => http://develpub.com/test_timezone.php

source code is more than minimal:

FIX IT DAMN IT

My site is still down (VPS)… figures the more you pay the lesser the level of service.

This is beyond excusable

Does a company that would let this happen seem like they have the foresight to get insurance for endusers? (rhetorical question alert)

I rebooted my Dedicated Server from the control panel and it seems to have worked.

Google: power-disruption-affecting-us-west-data-center-irvine-ca

“A description for this result is not available because of this site’s robots.txt – learn more.”

Way to limit damage by allowing us to vent in a controlled environment. Please just get us back online already. VPS still down.

@John: I’m sure I am not the only person who is caching this page. You can’t keep something this massive secret these days.

My boss has now asked me to find a new PS host for our organization. One with proper redundancies and backups. Joy. Now I have to endure a migration after all this, too. Bravo, DreamHost.

Update March 20th 8:00am: “…thank you for your continued patience.”

LIKE WE HAVE A CHOICE, DUDES.

Have put in ticket. Server is up but no admin access yet to o/s & s/w etc ?!?!

this was kind of exciting. finally something to destract me from real news. i hope no one was injured.

Mas de 15 horas sin servicio. Recien hace 3 horas volvio, pero se caen constantemente las bases de datos, va y viene y no se arregla nunca definitivamente!

March 20th, 2013 at 8:35 amKS2 Problema Says:

Not. Happy. At least 15 hours without service.

Either YOU will have to make REAL changes or I will. And I’m sure pretty much EVERYONE feels precisely the same — if they haven’t already started their search for new hosts.

You have a MAJOR issue on your hand — and it’s the second time in, what, a year?

I’ve been with DH now for a year and a half and there have been two major, extended outages. I thought the last one was inexcusable.

What does that make THIS one?

I’ll take your comments off the air.

I rebooted my VPS server with NO HELP from Dreamhost and my site is back up.
If you are on VPS try rebooting your server.

We’ve also been talking about finding another host. We’ve had several customers contact us so far with concerns.

DOESN’T DREAMHOST BACKUP THEIR POWER????????????????? it seems like a basic need.

March 20th, 2013 at 8:48 amcaptain reality Says:

I should start by saying that i’m equally peeved by the inconvenience it has caused me, but those of us in IT would surely recognize that this is not a trivial problem to solve given the large scale nature of the operation.

Threatening law suits seems more than over the top. I fail to see how you can claim you have a mission critical environment that now has lost revenue, yet you host it on a $9/mth service – seriously?? You have options out there like Amazon AWS, Rackspace, Heroku etc, but you went the cheapest option you could.

@DH – despite my appearance of sympathy, you need to step up your communication skills and customer service. The right thing to do would have been to pull the CEO out of bed and put some sincerity into your actions. It is pretty clear that this single event will cost you dearly.
Given you are now in damage control – you ought to think seriously about how to keep us.

@Sara – VPS reboot worked, thanks!

I reboot, site comes up. A few minutes later it’s down again. What the fuck is this shit.

VPS reboot worked for me too- thanks!

Why they would not recommend that is a mystery. Instead they say

“If you are still experiencing problems then your issue is no longer related to this power outage”

That’s really helpful!

My site is back down again. It’s been down for over an hour (after going down yesterday and then this morning again). Please help!

@Dan @John Glad I could be of help.

I agree, would have been helpful if DH advised us to re-boot our servers.

1, my website cannot be landed;
2, my website background can not login;
3, FTP can not connect, disconnect N,
I want you to provide server space certainly appear problem, may need to open the code conversion function, or open space, may be other reasons of server space, please help solve, now very worried…. Thank you

Ok! My website up! :)

Dan wrote: “VPS reboot worked for me too- thanks! Why they would not recommend that is a mystery. Instead they say “If you are still experiencing problems then your issue is no longer related to this power outage” That’s really helpful!”

That’s because like many people who are good at being insiders in an organization and poor at customer/outsider service, they only think about their own immediate agenda. Their agenda here is to get their system looking like it’s working from their angle, so they can get you to leave them alone by saying: It’s your problem now.

Of course, if customers are driven away by this, eventually there is a huge impact on the insiders, such as the business dying or contracting significantly. But they’d have to think longer-term to realize that, so nope. You’re on your own. Not their problem any more.

Looks like the websites are back but none of the files are there. We cant log in, its all gone. Is WordPress restoring our files?

Was down several hours, but able to rock and roll now. Sounds like the same sort of problem that Godaddy had when they were down for a day.

oops I mean to say: Looks like the websites are back but none of the files are there. We cant log in, its all gone. Is Dreamhost restoring our files?

Thanks Assholes. Just lost a client

March 20th, 2013 at 9:36 amMartha Smith Says:

Well I think I am going to try a hosting provider that runs off of solar power. That way my site will have predictable nightly outages. http://thinkhost.com/

March 20th, 2013 at 9:38 amEl papa John Says:

Ya funciona todo al 100% que buen susto, al menos nadie de mis clientes principales se dió cuenta. Y mi abuelita pudo facturar hoy. Ojalá se ponga las pilas el dreamhost en un futuro, no sería bueno que les pasara dos veces.

Martha,

The only problem I have with thinkhost is if it’s not windy, their throughput rates are as slow as molasses.

@captainreality — A lot of people are paying much more than $9/month, since Dreamhost also offers VPS and dedicated servers, ALL OF WHICH have been down this entire time. And paradoxically, DH worked on restoring shared hosting BEFORE restoring VPS, which is insulting.

After this is all over, DH should issue a heartfelt PR release where, in addition to an apology and promotional offers for customers who haven’t left yet, they explain exactly what happened, and what steps they’re taking to ensure that it doesn’t happen again.

March 20th, 2013 at 10:03 amJeremy Wright Says:

I have only myself to blame. I have known from the beginning that this is a half-ass company. I have learned my lesson.

Still down. Now nearing a full day. You guys suck. Seriously, how difficult can it be?

I pay an average of 75USD a month for VPS for the past 4 years. This is getting ridiculous.

I have yet to receive any communication from DH support, even though they are now telling people to contact support for assistance.

March 20th, 2013 at 10:43 amMatt Van Andel Says:

@Michael K.

I can recommend either Rackspace or MediaTemple as alternatives. Chances are that you are going to end up paying more, but those are the only other services that fit the bill.

DH

I would like someone @DH to explain to me why Shared hosting was restored before VPS hosting. I have to pay MORE to get LESS service?

I decided to reboot my VPS after getting no responses to my tweets or tickets this morning and Voila My site was up and running
Why did DH not send out a reply or post a comment asking all VPS users to reboot our severs instead of asking us to Wait?

My biggest question is why don’t you have BACKUP? An Ops readiness plan to mitigate such outages is a MUST to ensure a certain level of customer Service.

You brand your self as 100% up-time? not anymore! not if you do not have backup in terms of resources to back up that statement.

I hope DH has plans to compensate it’s users for the level of poor customer service and bad handling of this crisis.

Sara: I’ve had to reboot my vps about 8 times. It always comes back up after, but will go down usually a few minutes later :( Hopefully yours stays up a bit longer!

-Jamie M.

@Sara, they usually put shared hosting before VPS because it represents the larger portion of customers. I know, we pay extra we should be granted priority… but it’s probably best (financially) for them to lose few VPS owners than lots of shared-hosting customers.

We’re still unable to reboot our VPS (or any admin access). Help

“All other services related to shared, VPS and dedicated hosting are working and have been resolved for some time. If you are still experience any issues with your service, it is no longer related to this power outage. Please contact support so that we can assist you.”

Still have not heard from support even though we filed tickets.

OK. My site has been unstable for about a month. I’ve received shitty service by unqualified “kids” working their help desk. These past 24 hours may just be what causes me to leave. FYI … I’m down AGAIN.

Can anyone recommend a service that is more stable and has better support? Dreamhost is a joke.

Anybody seeing their dedicated server still offline?

We’re in. Rebooted and restarted our processes.
(Our inability to gain admin access was an external issue outside the DH network.)

We’re still experiencing issues with VPS. Reboot doesn’t work. No response from support, can’t even contact live help. Completely unprofessional and unacceptable.

It seems the vast majority of sites are back up. Several of my client sites and my company’s VPS are now back up and running.

I suppose it’s time to talk compensation. How does Dreamhost plan on rectifying this? Your max 10% reimbursement (aka: 3 free days of hosting) per your guarantee I imagine will not cut it with most users here.

@Sagi: I had to reboot our VPS like 8 times before it came back online. Keep waiting 15 min and reboot over and over. You may get lucky.

My site is working again. If it happens again, I will migrate… I have only had you guys for 2-3 months, but I’m giving this a chance that whatever you guys are doing with upgrades will help matters. Sounds like you need better backup/power configurations though.

Things are starting to work better. Ping times are back to normal, mySQL is finally working… slowly though! http://tafb.xxx/dreamhost_fuckup_fixed.png

-Jamie M.

If you’re a dedicated server customer and having problems – I had to SSH in and start httpd2 service.

Our site remains unstable. Here is what chat support just told me:

gary: They said to contact support and you have with support requests and via chat and all we can do since we are not the team that is working on it, is point you there.

When asked how much longer we will need to wait for complete recovery:

gary: i don’t know and they don’t

Here is the status about refunds:

hector: I don’t have any details on the refunds just yet. We are going to be sending out an announcement that will cover any questions that you may have regarding credits and refunds.

March 20th, 2013 at 12:36 pmrelax dont do it Says:

Been with DH for many years. Know the Irvine area well. The Irvine grid suffers lots of outages, if this is the data center near Main and MacArthur, then DH is not the only company that hosts there.

I know that the DH guys have attended VMware’s classes and certs because I’ve seen them there, so I’m wondering why they don’t have the SRM stuff working between Irvine and a redundant site in Irvine, say Level3’s building, or one of the many other Datacenter Buildings near the Spectrum?

Might be something to get VMWare’s consulting staff to help with, since SRM done properly would automatically failover to site 2 after the trigger threshold elapsed.

Now THAT would be 100% uptime.

Of course you gotta have some CAT backup diesel power out on the pad too. Who’s DH’s CIO…this should be his/her project with some VCPs.

Along with with commenters I would like an explanation of the priority order of service restoration.

I have been with DreamHost since ’99 and lived though the first big power outage in ’05 which resulted in DreamHost moving their status page to someone other than themselves.

I have moved from Shared to Multiple VPS to Dedicated boxes. Not to upset all those people paying <$9 a month, as somebody with a dedicated machine, why don't I get priority service. Sure restarting a shared server gets you more sites online, restarting my dedicated server gets you the same income as 15 shared web sites, so I am worth 15 shared hosting users.

I don't mind in parallel, but shared then VPS then Dedicated is upside down in my view.

What does Simon, Josh and Dallas have to say?

Yorick

Alas, poor Yorick, DreamHost knew him well.

Yay, I’ve been offered a $5 credit for A VPS and shared account that was down – so excited!!!

5 omnipotent Washingtons! Those guys have heart of freaking gold!

Just got an update that our dedicated server did have file corruption.

$5 USD? Wow, wonder how much that is in Canadian!! :)

-Jamie M.

file corruption. oops!

i’m *shocked*, **shocked**.

DreamHost Diner

Today’s special: Refried Data

March 20th, 2013 at 1:49 pmManuel Salazar Says:

Got back on line last night just before the next outage. Database is corrupt so all I get is a login page to our site. What now DH?

LMAO – This company is more like a comedy routine lately rather than a hosting company – it wouldn’t surprise me to see little monkeys dressed up in suits operating the VM’s and consoles. So many outages and so much down time – would love to see their % availability. DreamHost is a massive joke!! I cant even count any more of how many times my site has been offline with these jokers. Lucky I’m about to move hosts – then wont have to put up with their constant shite!

March 20th, 2013 at 2:19 pmEl papa John Says:

Es la salida del aire más fea que he visto, bueno una vez vi que blogger tuvo un apagón que hasta salió en las noticias.
Lo bueno es que no perdí ningún cliente, ubiera sido en otra fecha creo que me hubiera ido muy mal, hay que buscar alternativas. definitivamente.

Thanks for the update Simon Anderson, but I now need to apply for a credit? I guess I understand why you can’t do the honorable thing and issue credits, that would be expensive, but it would be the right thing to do.

… So liet me get this straight…

My ‘Green Hosting’ account with DH (more than half of why i choose dh to begin with) is now being run on DIESEL GENERATORS until these jerks get A PLAN FOR HOW TO FIX AN UPS??????????????!!!!!!!!!!!!!!

Absurd

GRRRR!!!!

going to (https://www.gandi.net/hosting/vps)

starting at $16

1 cpu core
512 ram
12gb
hd
~3tb tansfer @ ~10mb/s

they offer REFUNDS on service interuptions
they even offer they ability to return server shares you don’t need!!! (like, i can but 5 shares for the month, then decide i only needed two and return the unused resource time for a REFUND)

Amazing

I WANT A FREE T-SHIRT. LARGE PLEASE

This explanation is good enough for me. I just wanted to hear something about improvements, what exactly went on and what is being done… I really appreciate this, DH.

24 hour anniversary!!!! :)

“We believe, at this time, that Alchemy was performing unannounced maintenance on their UPS systems and the systems failed – resulting in a complete power outage. In addition to their UPS systems failing, their generators did not kick in.”

From Alchemy’s web site about Irvine data center – “Two power feeds from separate power grids”

So… Two power grids, “unannounced maintenance” (WTF) that knocked out all the power, fried internal routers (weird), and generators not kicking in. Why didn’t they just start the generators when it went down initially? Doesn’t seem like a very good back up plan for a major host.

These outages are happening very often… maybe my server will be faster since everyone is jumping DH’s ship :(

Anderson’s letter just disappeared again when I refreshed the page just now. FWIW

Now the letter is back again. Not sure if it’s identical or different now…

haha, letter has no formatting now.

They probably went to update with the status report and messed up the copy/paste job I guess?…

Yes!! as always, blame the building, blame the array

I can’t fucking read that for shit, put some break lines and some formatting. Basic stuff here no wonder the power went to shit.

OH and I smell bullshit, just a tad of a cover up really.

So blame lies with Alchemy Communications.

AND YET “DreamHost owns a large equity stake in Alchemy Communications.”

(http://www.alchemy.net/)

“Alchemy’s newest data center in Irvine, CA, is a state-of-the-art, 42,500 sq. ft. facility located near John Wayne Airport. This carrier-neutral facility is equipped with every essential to house and protect your data.”

LOL funny that. I wouldn’t call it state-of-the-art… but OK.

Still in tech support hell:

GregB: I have also noted that the homer server load is quite high at the moment. I have contact our team and have asked for further assistance.

Sadly, load has been high since 1:00PM and probably earlier. Still have not heard from a real person. Dean Johnson’s email does not count.

Your guaranteed uptime policy is a joke. You know your system went down, yet you require a support ticket be submitted to receive any credit.

Do the right thing, and give everyone a 10% credit on their next bill no questions asked.

Your lack of control over the situation is very troubling. I want to deal with Dream Host NOT Alchemy Communications!!
My team is actively searching for a new responsible company to handle our service. The damaged caused by this to our business and brand are reprehensible.

“All customers impacted by these service issues can apply for credits or refunds in accordance with our guaranteed uptime policy by contacting support through the panel.”

$20 refund? for VPS being down for 24 hours?

Pack your shit kids we can’t live here anymore, daddy doesn’t love us.

Too little too late Simon. We are moving so we can start a new life with someone who cares about us.

-Bail

March 20th, 2013 at 4:42 pmJames Printer Says:

Aaron: Where are you going to go? Amazon certainly don’t care. I’d suggest Linode.

March 20th, 2013 at 4:52 pmDavid Bounds Says:

These down-times are unfortunate but understandable. Thanks for the the completeness of the reply. My sites were back up quickly.

March 20th, 2013 at 5:12 pmSuitspeek 101 Says:

For those who can’t translate Bullshit:

The in-line UPS system failed, breaking power from the grid.
The backup generators were not started up.
The failed UPS system was reset instead of repaired and we all crossed our fingers.
The UPS system failed again. Just like it did the first time.
The backup generators were started up manually. Under advice from the tea lady.

Down again. :(

LinkedIn went down for a while this morning. I wonder if it was caused by the same outage.

My stuff never went down but I’m in the LA DC. Dreamobjects came up sometime this afternoon.

UPS failures are every IT/telecommunications company’s worst nightmare. That equipment doesn’t like being turned off.

Unannounced maintenance on a UPS at a datacentre is bad procedure.

If you run a WordPress site with decent traffic, consider moving to lightningbase.com. After this issue with Dreamhost I switched to them today – just finishing up now. They transferred all my sites and basically did all the work for me, for less money than I’m paying for my Dreamhost VPS. They’re a small company but have two data centers and have redundancy built in to avoid these sort of situations.

“All customers impacted by these service issues can apply for credits or refunds in accordance with our guaranteed uptime policy by contacting support through the panel.”
Did that yesterday, but just keep getting the same generic reply spammed…which makes me even more angry.

@Patrick Delahanty – apologies on that, when a major support incident is occurring we have to mass reply. If you send in a request now we will make sure our crediting team gets a hold of the message. Say Ralph sent you.

Website spsfpulaski.org still down, submitted ticket to support 10 hours ago but haven’t even had a reply. We have a major fundraising campaign starting tomorrow and no website!

@Daniel – getting my support team on your issue right now.

@Daniel – IP related issue, we are working on fixing it now. We will contact through the support email system once we have the issue resolved. Actually I just checked and the site is up now.

March 20th, 2013 at 7:49 pmone of DH user Says:

So DH think they can pass all responsibility to Alchemy to dodge blame?

I would really appreciate a personal response to the issues addressed in my ticket from DH rather than multiple generic messages.

Systems fail I get that – my issue is with how poorly the recovery was handled.

NO I will NOT be submitting another ticket to get a credit, DH needs to do that automatically for your customers for the service interruptions caused!

“All DreamHost systems and services in the Irvine DC are up and stable, and have been since early this afternoon PDT.”

Not true. Chat support keeps telling me that the Homer’s load is high and there is nothing DH can do about it. My website is still down since yesterday afternoon.

@one of DH user: We are ultimately responsible for your uptime. Our status post was to inform you on why the sudden and unannounced outage occurred. This is why we are taking the steps, with our data center, to make sure that the possible upcoming emergency maintenance is a success.

@Frustrated – Status post updates are strictly related to the issue. Server loads are not caused by the power outage and can have a number of reasons why it is happening. If you link your domain or provide it in a post I can get someone working on that.

@Sara – Please list your domain so we can get your account credited and answer your questions directly. Thank you.

@Ralph ticket #5666816

@Ralph sorry here’s the ticket with my concerns # #5665862

March 20th, 2013 at 8:30 pmJeremy Wright Says:

HA, that’s funny, it almost seems like Dreamhost is trying to sound like they know what they are doing.

MongoDB shell version: 1.6.2
connecting to: admin
Wed Mar 20 20:22:05 Error: couldn’t connect to server 127.0.0.1} (anon):1137
exception: connect failed

Any hope of getting this mongodb online- I’ve been twiddling my thumbs for over 24 hours now. Really need to get this up. Please!!!!

@Sara – I have one of my leads responding to you.

@Barry – have you submitted a ticket? I don’t believe your issue is maintenance related. :) I can get someone to help you though.

I would like to know why we have to “apply” for credits. At this point, we are all trying to recover from the disaster, and having additional work of “applying for credits” is just adding to our burden. You know our sites have been down! You can apply the credits automatically!
Even those small credits do not really adequately compensate us for what has occurred.

@Roy – My support team will be in touch.

@Ralph

I just finished chat with Hector. Here is what he said.

hector: “Okay, so there were some issues with the host machine earlier today that was related to the outage from yesterday. Those do look to be resolved. We did email you back in regards to that earlier today. We also did notify you that we had your ticket with our supervisors in regards to the credits/refund you would receive for the downtime.”

Yes, I did receive one month’s refund for the 24-hour downtime around 1:00PM, but that was approved by a chat support person. After Dreamhost’s CEO sent out that mass email, my site was still down and I could not log into my VPS. When I contacted chat support, all they could tell me was that Homer was overloaded and that I had to wait for the special systems engineer to resolve the issue.

Hector did a reboot and got my website up. While I am happy the site seems to be up now, I am not sure it will stay up given the troubles I have had, since that misleading statement from the CEO. What chat support kept telling me was to file support tickets, but at no point did anyone from that special team contact me about my tickets. According to Hector, the load on Homer seems to be back to normal now, so that’s why his reboot worked. Well, if the special systems engineers were able to resolve the system on Homer before I got a hold of Hector, why didn’t they just reboot the system for me? Why didn’t they reach out to me, so I could reboot it?

Here are the tickets I have opened on this issue. I would love to hear from a real person about these tickets.

#5662152
#5665562
#5666669
#5666674

@ralph
thanks for getting back to me.
Yes I’ve submitted a few but they just auto respond pointing me to this page.

@Frustrated – getting a lead to address your questions.
@Barry – ticket ID and I can get you some help. Unfortunately with large support incidents sometimes non-related requests get mixed in. I sincerely apologize about that. Will try to get you some help soon.

I sent emails to support@dreamhost.com several times yesterday but never got a response. Guess everyone was busy fixing the problem?

My site was down for several hours yesterday, past midnight for sure.

I understand you’ve committed to refunding everyone who was affected.

How do I get a refund?

@Sasha H,

What I did was to open a support ticket under Billing. You should get one month’s credit.

@Ralph
Here you go

#5665257

Thanks Frustrated :)

@Sasha H – have my team contacting you based on the domain you linked.

@Barry – I have one of my leads on the case!

@Ralph – Thanks I did get a response

I need to wait 1-2 days to get a credit now?
“Please be on the lookout
for an email regarding your credit in the next 1-2 days.”

@Sara – I authorized a credit. Sorry about that. We do have a special team handling the credit but I approved yours now. You will get another response.

Thanks @Ralph. Got this automated message: DreamHost Support Ticket #5666472 Has Been Moved. Look forward to an explanation from the special systems engineers about the downtime after Dream Host CEO’s letter went out.

@Ralph – Appreciated

@Ralph Thanks I got the refund email.

Had one of your support team been here responding to queries earlier, you would have had less frustrated customers – Better Late than never :)

@Frustrated – I still have a lead looking at your requests. That auto message was related to the ticket, but I do have someone checking for you directly.

My tracking number is #5667067

@Sara – it is my policy to have people chatting in here, with large support incidents with a lot of moving information it can be a bit of a bear. Thanks for the input! I’ll try to come up with a better system for handling posts on high volume comment incidents.

@Ralph – status updates are great but a personal response even to say “We are working on it” does wonders #justsaying

“Hello,

Your inquiry has been moved to the queue of a specific tech support team member (this is either because they are already familiar with your case or are the best equipped to assist you with this specific issue). They will respond to you as quickly as they can but depending on the complexity of the issue it may take longer than normal for them to get back to you (even in excess of 24 hours in some cases).

Thanks!
The DreamHost Ticket Moving Robot!”

Hi Ralph, do you know when a HUMAN BEING will be addressing my refund request?

Sasha

@Sasha H – I have someone replying to you.

THANK YOU Ralph and Elizabeth :)

“Hello,

I am very sorry for the problems and the downtime. As an apology, I’ve
credited you with an amount equal to one month of hosting, USD $XX.

Thanks!
Elizabeth L”

@Ralph
Brian-A Sorted the Mongo problem for me. Hallelujah.

Keep up the good work!

@Ralph, finally got a real human response from one of your team members. Thank you.

I’ll repeat what I told Brian here:

Thank you, Brian. It is good to get a real human response on my ticket. Yes, Hector did reboot the server, and so far so good, fingers crossed. I am still not clear why no one from the special support team told me the issue with the overloaded server has been resolved or why a reboot was not done on my behalf, especially after being told by chat support prior to Hector that I had to file support tickets. I’ve done plenty of reboots earlier today with little success, so I had no reason to think another VPS restart would solve the issue. It is that lack of communication, coupled with my rereading of your CEO’s letter earlier in the day, that motivated me to start posting to the status page more consistently. I know what Ralph means, but when my site is still down, it is a slap in the face for the CEO to say “All DreamHost systems and services in the Irvine DC are up and stable, and have been since early this afternoon PDT.” It is also frustrating to have the status page and chat support direct us to file support tickets yet with no follow-up messages from real humans from that special support team explaining what the problem was and what steps they were taking to resolve the issues.

Here’s a statement from DreamHost:

——–
“We are all really bearry sorry about the extended downtime … from the … power outage!

The power was only out for about an hour, but as it came back on, there was trouble, trouble, trouble. Our router started acting funny, some file servers were mis-configured, some web servers didn’t want to come back on, and so on, and so on, and so on…

Although most things were back up and running within the five hours, the network in general was still flakey for about 8 hours…”
——–

But guess what? This statement wasn’t from today or yesterday. It was from February … 2007! It sounds like this exact thing has happened before, and they haven’t learned from it.

http://dreamhost.com/dreamscape/2007/02/28/super-lame-apology/

@Frustrated – I fully understand. Large support incidents put us in a bit of bind, man power wise. We continuously try to learn from these incidents to further improve our response to these type of incidents. Truly sorry for the frustrations and your feedback is appreciated. We always can improve.

“Special Support Team” bwahahahaaa :D

“Special Systems Engineer” bwahahahaaaaaaaa :D

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency maintenance on DreamHost.com, FilesForever, Whois, DHurl, DreamHost wiki, and Discussion Forums March 19th 12pm PST (RESOLVED)
11:20 am

Emergency maintenance on DreamHost.com, FilesForever, Whois, DHurl, DreamHost wiki, and Discussion Forums March 19th 12pm PST (RESOLVED)

Posted (March 19th, 2013 at 11:20 am PST) by Ralph

At noon today we will be conducting hardware maintenance for the server responsible for the front end of DreamHost.com, FilesForever, our whois proxy, DHurl, Discussion Forums, and our customer wiki. This maintenance is to prevent possible unplanned problems with the server today. The maintenance will start at approximately 12pm PST and should last no longer than 1 hour. Our apologies for those using these site tools and the FilesForever service. We will update this post periodically as we progress through this short maintenance.

Update March 19th 12:05pm: This maintenance has now started. The services listed above will be out for a short period of time. If we encounter any complications we will update this post with further information.

Update March 19th 12:40pm: This maintenance is now complete as of 12:28pm. Thank you for your patience.

This entry was posted on Tuesday, March 19th, 2013 at 11:20 am and is filed under Data Center Maintenance, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Emergency maintenance on DreamHost.com, FilesForever, Whois, DHurl, DreamHost wiki, and Discussion Forums March 19th 12pm PST (RESOLVED)”

I don’t know about these status… but I know one thing that my websites are hell slow and now working really good…

Please solve this issue

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » POSTPONED: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 20th 8pm PST (4 hrs)
5:31 pm

POSTPONED: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 20th 8pm PST (4 hrs)

Posted (March 18th, 2013 at 5:31 pm PST) by andrea

We are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May 2012 is finally nearing completion. We started this project when we realized our network stability was in danger as we had outgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, March 20th, we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail including Webmail, MySQL, the Web Panel and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update March 20th 2:00AM: Due to downtime caused by our Irvine datacenter outage, we’ve opted to postpone this maintenance. Once we are able to reschedule this, we will re-post this notice! Thanks for your understanding!

This entry was posted on Monday, March 18th, 2013 at 5:31 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Wednesday March 13(RESOLVED)
12:42 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Wednesday March 13(RESOLVED)

Posted (February 25th, 2013 at 12:42 pm PST) by andrea

On Wednesday, 3/13 at 8pm, we will be performing a core router software upgrade in our US-West Data Center (Los Angeles, CA). We are scheduling a 4-hour maintenance window to safely test and monitor all the steps needed for this maintenance. During this upgrade, the network traffic will flow through temporary and thoroughly pre-tested routers, to minimize downtime.

This core router software upgrade is necessary to move on to the last step of the network maintenance in this data center, which has been previous postponed:
Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)
We will post an announcement once we are ready to go ahead with this final step again.

As always, please contact support if you have any questions. You may also check if you’re affected by this maintenance by looking up your data center location on your DreamHost panel.
We will update this status post as the maintenance moves forward and will be monitoring comments as well at that time.

Update March 14th 12:55am PDT: We have successfully completed the core router software upgrade. All network traffic is now going through the upgraded and tested routers. We will continue our network maintenance with the final step soon — look for an announcement on this status page!
As always, please contact support if you have any questions.

Update March 14th 12:01am PDT: The core router software upgrade has been successfully completed. We are moving network traffic back to them as I type this. As expected, so far, there has been no customer impact from this maintenance. However, to be on the safe side, we’re expanding the original window by an hour, so we can complete all necessary changes and tests. We will continue updating this post as we progress. We appreciate your patience.

Update March 13th 11:05pm PDT: The switch to the temporary routers has been completed without a hitch, and we are currently running the software upgrade on our core routers. Once that’s finished, we will be directing network traffic back to the upgraded routers. We will continue updating this post with our progress, and appreciate your patience during this maintenance. Again, we don’t anticipate any customer-facing issues, but please contact support if you have any questions.

Update March 13th 10:05pm PDT: We are proceeding with our maintenance as scheduled, without impact on our customers. We will continue updating this post with our progress.

Update March 13th 9:05pm PDT: One hour into the maintenance, so far without any customer-facing issues, as expected. We’ll continue updating this post!

Update March 13th 8:35pm PDT: So far so good! We’ll continue updating this status post with further progress reports.

Update March 13th 8pm PDT: We are going ahead with the above maintenance as scheduled. We will be updating this status post as we’re progressing.

This entry was posted on Monday, February 25th, 2013 at 12:42 pm and is filed under Data Center Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Wednesday March 13(RESOLVED)”

Can you add dreamobjects to the location list in the panel? well this be bringing dream objects down?

Phil, good suggestion, I’ll pass that along! And no, the DreamObjects will not be affected by this maintenance, they are in a different data center.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network connectivity issues at US-West Data Center (Los Angeles, CA) – ( RESOLVED )
11:52 am

Network connectivity issues at US-West Data Center (Los Angeles, CA) – ( RESOLVED )

Posted (February 22nd, 2013 at 11:52 am PST) by brians

We are currently experiencing network connectivity issues in our US-West Data Center (Los Angeles, CA). This is resulting in site downtime. Our network engineers are working to restore connectivity. We will update this post with further information as it becomes available.

UPDATE February 22nd 12:45PM PST:: We are happy to report that this outage has been resolved. All services are up and fully functional. Thanks again for your continued patience.

UPDATE February 22nd 12:33PM PST: Our network engineers are continuing to monitor the situation. We will resolve this post when we have confirmed that all services are back online again. We sincerely apologize for any problems that you have encountered. Your continued patience is greatly appreciated. -Brian S

UPDATE February 22nd 12:12PM PST: Our networks engineers found the cause of the outage. We sustained a power loss to one of our racks that held a distribution switch and several servers. We have restored power and are investigating how to improve our alerting systems for the future. Please accept our apology for any inconvenience this has caused. -Brian S

This entry was posted on Friday, February 22nd, 2013 at 11:52 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

25 Responses to “Network connectivity issues at US-West Data Center (Los Angeles, CA) – ( RESOLVED )”

This is the second time in 24hrs,

Number 1 / 3:14 PM (19 hours ago)
Over the last 24 hours, Googlebot encountered 1 errors while attempting to connect to your site. Your site’s overall connection failure rate is 33.3%.

Number 2 / NOW
Error establishing a database connection
Database Error: Unable to connect to the Database: Could not connect to MySQL.

100+ client’s websites are down, this becomes a pattern…

First time in, um, 108 days.

A lot of outages this week, this was different from previous but the end result is the same: users can’t get to my sites so they lose confidence. And I have nothing to tell them most times because this site rarely updates the outages, and is cagey for a while even when it does. Please address the underlying problems.

Guys–

I have news for you. These outages happen all the time, and they have been happening at this frequency for years. Every time one happens, people get pissed and threaten to leave. But they rarely do leave. The reality is that with any shared, non-entreprise level web hosting service, you’ll have these outages. It’s just part of the deal when you’re paying something between $4.00 and $400 per month. If you want true reliability and uptime guarantees, you need to pay through the nose. I’m not quite sure why Dreamhost doesn’t have more redundancy built in between its data centers (I mean, why should an outage at one site really bring down any cilent sites, given that there are other data centers that should theoretically be able to pick up the slack?) but my guess is that it has something to do with the cost to provide that kind of redundancy. The truth is that these outages will continue, and the vast majority of customers won’t budge. Even if Dreamhost loses a few customers here and there, it’s still way more cost-effective for them to lose those customers than it is for them to implement the kind of infrastructure that would make downtime like this not happen. In reality, there aren’t other “budget” hosting services like Dreamhost that provide any more actual uptime, so there’s nowhere better to go unless you’re willing to pay a ton for it.

The outage or anther issue is still causing significant issues with my site.

MySQL server is not responding

all my databases are still down!

My sites are still down.

My websites are down.

Mine’s down now, been intermittent past few hours.

It’s not possible….very bad service.

My WordPress installations are all affected since they are timing out trying to connect the MySQL. MySQL takes minutes to access if they don’t time out. Customers are upset.

Outages are not a problem at all. Quite the contrary. It gives that charming human feel to what we call “technology”. Love thy server ;-)

Ummmm.. don’t think this has been resolved as I’m STILL receiving a boatload of connection errors :(

Guys, what the heck is happening? We have a live event with 1300 people starting in 36 minutes and your servers have been down for more than an hour now! You must be kidding with your service, it’s the freaking 6th time in the past few months when this is happening and I don’t find any excuses anymore. Can you please get your stuff together and get it working? If our website will not be up in the next 30min we’re facing thousands of euros in damage! Please let us know ASAP what the heck is happening – support@brandonbays.tv

We are down too. And hard. WTF!

why the hell does it say “resolved” in the title when we are all still down?

my websites still down!

My message board was hard to load a couple of days. It was not resolved at 22nd…. The problem prevailed until Sunday 23rd. I hope it is better today.

I’m using cloudflare, so front end is served by it. But couldn’t log in to my wordpress sites to maintain.
Is this issue really solved?

What the hell has been going on the last few days? Frequent downtime! I’ve had enough of this shit. I don’t now how a web host can have so many outages. If I was more tech-savvy I would have moved my site off of this shitty host long ago. I guess I am going to have to figure out how to, since you guys are killing my site.

Database isn’t working. Web sites are down!

My MailPro app on humorlabs.com has been down for two days now. What’s going on over there?

vps barriodelcarmen down
ffffff

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DNS Server Problems ( RESOLVED )
12:20 pm

DNS Server Problems ( RESOLVED )

Posted (February 14th, 2013 at 12:20 pm PST) by brians

We are experiencing some service interruption as a result of an issue with our DNS servers. This may cause site downtime, slowness, etc. Email could be impacted as well. We are working quickly to resolve this and will provide an update as soon as possible.

UPDATE February 14th 2:04PM: The DNS servers remain stable, and we’re continuing to monitor but consider this incident resolved. Thanks again for your continued patience.

UPDATE February 14th 1:58PM: The DNS server have continued to remain stable. Our System Engineer Team continues to actively monitor the flow of traffic. However, we do not want to make that call until we are 100% sure. Thanks again for hanging in there with us.

UPDATE February 14th 1:29PM: The performance on the DNS servers have improved dramatically. All services ( site, email, etc ) should be operating at optimal performance. Our System Engineering team is continuing to monitor the stability of the dns servers. Everything looks good from our perspective. We just want to make sure that issue is indeed completely fixed prior to resolving the status post. Thanks again for your patience.

UPDATE February 14th 1:07PM: All 3 DNS server are now responding. However, query times are lagging a bit, which may still be causing some slowness to your sites, email, etc. Our System Engineering Team will continue to work towards resolving this matter. Thanks again for your patience.

UPDATE February 14th 12:49PM: We believe that we have found the source of the problem. Our system engineering team is currently running tests and diligently working towards a solution. Thanks again for your patience.

UPDATE February 14th 12:27PM: Our system engineering team is diligently working towards a solution. Thanks again for your patience.

This entry was posted on Thursday, February 14th, 2013 at 12:20 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

150 Responses to “DNS Server Problems ( RESOLVED )”

Thanks for posting this fairly quickly. I’ve been seeing downtime for about 5-10 minutes on my sites, and just figured out that the issue was DNS and was going to submit a support ticket.

My sites have been going up and down since 5am this morning (CST)

February 14th, 2013 at 12:22 pmGerman Hernandez Says:

Damn. I was getting used to have a flawless service

Hopefully this can be resolved soon. Really shakes the confidence of my users to see the “Error establishing a database connection” splash page. Thanks for looking into it.

They will call this: “DNS Storm Nemo.” Casualties are expected to be in the thousands of visitors.

HELLLO There,

I just released my new template and it is the first to host here and now i don’t have preview and i am loosing my theme now, Please fix this as soon as possible.

Yep – my site’s down. Why isn’t the Cloudflare cache working for this site? I thought it was supposed to serve up the cached pages even if dreamhost servers went down.

Ah, this would explain why I can’t get any mail. Thanks for the timely notice, and hope y’all get it it sorted easily.

They have been having issues for a few days, finally got a K.O punch. Hard to make money when i can’t work on a webbsite for a client..

Would this be the cause of a “Error establishing a database connection” message for one of my sites? It seems to be only for that one though my other sites hosted here are ok.

haha typical i have literally just signed up to dream host

February 14th, 2013 at 12:25 pmMiguel Chateloin Says:

Thanks for keeping us informed. I hope you get this resolved soon. I’m developing a website for a client and I really need my testing site to conitnue working.

This is unacceptable for us. After months of work we finally had some site content getting a spike in traffic from major social media sites today. Please let us know when it is fixed.

Whammy. Just setup my first site with you guys and the thing goes down 2 hours later. Bahaha oh well, I’ve used your service with other clients in the past no issues with them. We’ll see how this goes. Thank God for free trial periods.

After a flawless trial period of a week, this happens on the first day I pay! My site’s unreachable at the moment.

DreamObjects down! :s 100% up time ? bullshit

Error establishing a database connection. I thought site got hacked first.

Hope you can fix that as fast as possible, good luck and happy st. valentines to everybody!~

Thank you dreamhost. My site is down but I thought it was due to my upgrade to VPS yesterday. I recently migrated from somewhere else and I am really happy with your communication.

so disappointed right now :(

Also experiencing issues on one of my three websites hosted with Dreamhost (on the spica server).

Imagine if today was Valentine’s Day, one of the biggest online sales day in the year!?

I really hope you get this resolved quickly. After the almost 24 hours of downtime my sites had last week, my tolerance and that of my clients is wearing thin. Curious, my dedicated client sites are fine (well one is using outside DNS but the other is using DH) are working fine. Why is that?

Bloody Murhpy! Completed set up our first WP test site 20 seconds before outage! Is there a chart of historic reliability/uptime?

Well this is fantastic… on the day my client is having his test run we lose service.
Sigh.

thanks for the update!

I am new. This is a bit annoying, as I thought that I did something wrong. Is there a possibility to send an E-mail, when the service is restored?

Back up. Thank goodness.

This sucks, i feel like i’m on a overcrowded server. I’m always having trouble, previously it was excruciatingly slow – now it’s down totally!!!!

Sucks!!!

Yah, it’s back up!!! Thanks for looking after that so quickly. We had just announced out new site to potential clients so I had a mini heart attack!

Was just talking the powers that be out of making me switch from dreamhost to a local business this morning. The past few days’ uptime is not helping.

I really appreciate the backend options dreamhost offers, but the uptime is getting terrible.

It looks like things are back up (at least for us). Fast resolution. Well done.

When my sites go down, i lose visitors which means i lose money. Then i lose my position in Google search ranking, which means i lose more money. My clients servers go down as well, which means i lose much more money. I am experiencing this with a well paid dedicated server.

I chose dreamhost because I thought it was reliable. I am not here because it is cheap. If you cannot provide proper service, please let your clients know. I experience 4-5 issues a month. This service level is almost same as a cheap shared hosting.

I’ve not been able to send or receive mail for a good while now.

This is unacceptable.

Really folks, be a little patient. This only posted at 12:20pm and already you’re castrating them? If this was GoDaddy, they’d lie about the DNOS attacks from Anonymous and say it was something entirely different.

Not up for us yet :(

Breaking my heart DH. After 8 years, I think it’s time we parted ways.

why is this affecting email that is handled by google servers? that has me confused.

Nothing but problems since I have bought the host….. Bought them sunday site was down 2+ hours on monday and now down again….

Before moving to them i was using my own server/network and i have about 2000+ page views a day and my network never once went down…

Site starting to get big thought it would be a good idea to get on an bigger enterprise system….. Mby i was wrong.

Back up! Thank you :)

Was terrified when I got the Database error message. This makes me feel better, I think? Can anyone in this forum tell me how often this happens? How long to expect an outage? what to tell my (non-technologically friendly) boss when he wants to know what is happening – lol?

Is this still a problem for anyone else?

After an 11 day outage is just been recovered 3 days ago, now this…. great work guys… I am out of here as soon as I can move everything…. not a dream…a nightmare….

I think people are upset not because of this one particular issue, but because this is becoming a pattern with them. And unfortunately, the world will not be patient for frequently slow/down websites.

We are still down… Getting Database Error.

My site is still down. Can we get another update? Or estimation on when it when sites will be back up?

Still not up here…database connection error

If it only took 15 minutes of downtime for the “unacceptable” comments to come from some of the users here, why don’t you guys start hosting yourselves? Or get a local/ISP to host your stuff with business assurance? You guys sound really dumb.

Il mio sito non risponde !!
http://www.duriibanchi.com

We are still down(“Server not found”).
“boltonia” server :(

Please fix it fast. There has been to many downtimes and interuptions on service in the last days. Difficult to work with our clients taht way.

As it states above entry window, “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

our domains and subdomains are still down, Julia. The hosting for one of our domains is working (MySQL connection isn’t available for that one), but the others are still totally down.

Hi

Jim you are lucky if you experienced issues this only today because I am having issues almost every 2 days: slow, for some days it is impossible for me to use phpmyadmin as it takses MINUTES to load tables etc.

So i am sorry to say but i really do feel disappointed :(, it looks like this is a server used for “second zone clients”…

Excuse my English i am French!

We had several adwords campaigns active, I guess we are losing the money…Dreamhost should be worried, their service is getting worse for the last months.

Hmm… I am all for understanding issues on your side, but when you blatantly advertise 100% uptime, I expect that to be kept to. It’s the top reason I chose you guys. Literally just launched a website and it now looks completely unprofessional with MySQL issues. Can we expect some kind of compensation? Doubtful…

Disappointed.

Thought I was having an issue with my CloudFront or S3 buckets. Was really starting to scratch my head. Thank you for posting. Tough that we literally just began a media push on a now-unaccessable piece of media this AM.

To All – I have been using Dreamhost for 7 years with very few problems. Sometimes these things happen and it’s never going to be a convenient time for anyone. I’m sure this will be over very soon and your visitors will try again, if they cared to buy from you in the first place. My site is down and so are many of my clients. I sent them an email for damage control and let them know that this is rare and to rest assured that the folks at Dreamhost are doing everything they can to resolve this. I’m sure it’s a mad scramble at DH right now, so don’t get so upset with them.

For the record… I monitor the uptime on all of my sites and it remains pretty steady at 99.8%. I think that is a pretty good rate. You want 100% with tons of redundancy… you better be prepared to pay for that. Dreamhost is affordable, responsive and an all around great hosting company.

Anyone shopping? Who is a good alternative to “DreamHost”?

Our site just went down as I was making a change. Scary! Thought I had really messed something up.

Bob, Jim True – it’s not the first time!

After 2 years of hosting with Dreamhost, we’ve had more than a dozen outages, some of them for over 24 hours at a time. This is the most unprofessional and unreliable web host I’ve ever dealt with in my 15 years of web development. It’s a real shame, because they used to be really good, and now they’re just circling the drain.

Be patient folks…I’ve been with Dreamhost for years and downtime has been quite rare. They’ve published information about the downtime. They know and care that your site is your income. They are working diligently to fix it. What more could they reasonably be doing?

Well, I can get back on the site but I am unable to upload any images “failed to upload due to an error” That’s how this outage started for me.

@Brady Mills
I keep track of uptime as well, with Pingdom, our homepage has had 99.45% over the past month and our DB has had 98.37%. That’s not something to brag about especially considering they “upgraded” their network. Plus, this doesn’t take into account the very slow response times that have been building up over the past month.

Nothing working for me at the moment. Will dreamhost refund me at all? Nope.

Since December it has been slow and unreliable almost weekly.

Still getting Server not found on almost all domains hosted here. This comes at a time when Advertisers are buying ad spots from me. Not good.

Can we have a ETA on this before webmasters start pulling back links on blog trades from me and others?

Thank you.

We’re back up after 14 minutes. No fun, but stuff happens. Onward…

And we’re back up. Not too bad, wish it had occurred at a different time of day though.

Thanks for getting it fixed. What was cause and will it happen again?

Just get the frikkin thing up. It’s not that frikkin complicated. Do your job.

Add our site to the list of those for whom this issue has now been resolved. This is the first issue I’ve had with DH in about a year and a half of using them, FWIW, and I’ve appreciated their service.

I’m getting database error connections for one reason or another an awful lot of times this days… What’s going on, Dreamhost?

@Brady Mills – where can i get some good website uptime monitoring?

Will you be refunding VPS costs since this crippled it’s usage?

My site is back up. Nice work Dream Host.

Thanks for your promptness in addressing the issue. Still, as many mentioned, we expect the best. Considering a move to AWS unless some level of consistency can be achieved.

My 8 sites on DH are now back up. Still a fan of DreamHost. Thanks for the quick resolution guys!

Unbelievable. 2nd time this week.

I’ve been having this issue for about an hour and had sent off a ticket already…

*edit* I was just going to post this comment and now my site is back up. Thanks DH!

Thanks for the update. Kinda had a mini-freak out when I saw the “unable to connect” across my screen. Hope you guys get it working soon!

i know now why my adsense earnings have been going to recently :( Wonder if i should go back to 1&1 :(

Hey, patience is a virtue. All my sites are coming back up. I have been with Dreamhost for years and they are good about uptime. They are also good about communicating and then getting the issue fixed. All will be well.

Nice recovery, thanks guys. My client was giving a demo to 10 ppl and with you guys putting up this post and a quick reply from the Support chat I was able to reassure them until service could be restored. Appreciate the expediency!

We’re back up now. Thank you.

Please, all my sites are down

Work faster, you monkeys.

My sites just came back online.

Delivery to the following recipient failed permanently:

you@yourdomainondreamhost.dns.com

Technical details of permanent failure:
DNS Error: DNS server returned answer with no data

We are back again, thank you for the fast resolve.

I am up again too! Thanks!

some of my domains are resolved. I too had to assure some of my advertisers by using this page.

thanks.

It’s really dishonest to make claims like ” Our 100% up-time guarantee means we take up-time seriously” when you really mean “WHEN we go down and it will happen surprisingly often, and if you specifically contact us and insist, we’ll repay you for the time you were down, but since you pay so little for hosting, it’ll amount to pennies from us. Again, we won’t just refund you or anything. You have to actually contact us and insist we refund you for the down-time. This is the same thing as 100% up-time, no?” No, it isn’t.

I don’t WANT a refund for downtime. I want what I expected when I signed up: 100% uptime. Or at least something close to it. That’s what you claim on your site.

I have several big outages with my shared servers, VPS servers, and dedicated server every year. Several. How can you claim “100% up-time guaranteed!”?

If I had one complaint about Dreamhost, even more than the continual down-time itself would be the lack of integrity in how you blatantly lie about 100% up-time.

I like Dreamhost – they help me continually but I do have to say that since moving to a VPS, I am still having issues on a site with very little traffic – how can I be out of memory so many times per day and expect to grow.

I think their staff is absolutely terrific though.

my site’s front end is up, thanks to WP Super Cache. however, I can’t post an update now since I’m still getting “Error establishing database connection”. My cache expires in 6 hours. Hope this will be fixed before it expires, or my site will totally go down. :(

Second time I have had problems in the last 7 days.

3 yıldır kullanıyorum ama artık bu sorunlardan çok sıkıldım, devamlı problem,artık daha az sorun çıkaran bir sunucu arıyorum :(

You guys are amazing! There’s barely any downtime, but when there is you’re always quick to try to resolve it. Just wanted to say thanks for that.

Using for 3 years, but now I am so sick of these problems, continuous problems, I’m looking for a server which is now less of a problem: (

Can you confirm if emails sent during the down time will eventually be sent out? If not, what is the time period during which no emails could be sent so I can resend those important emails? Thanks.

Oh, and worse yet, your tech support people will typically write off any problem in the easiest and quickest way possible to get the ticket off their plates. It happens like this:

1) You send in a support ticket because something’s jacked up again. Probably because you were spontaneously moved to a new server without notification. They’ll say they notified you, but you usually don’t get the email they claimed to send you.

2) Serveral hours pass (At least six, but often more) and a tech gives the ticket a cursory glance, glances at your server, makes a quick, simple assumption about the problem, doesn’t confirm it or look into it any further than that. Tech emails you back with “Yeah, the problem is: [insert blame about something you’ve done/changed/need to increase resources on server].

3) You email the tech back with extremely basic and obvious details about why they are not correct that they would have determined on their own had they put even the slightest effort into looking into your issue. The more time you waste, spelling everything out here, the more likely you’ll get someone to actually look at the problem. You have to essentially do most of the investigative work yourself and FORCE them into the problem with so many details, that they can’t write the problem off with a lame, unrelated accusation.

4) Someone else entirely gets the ticket, reads your response and: Several more hours pass before the ticket is picked up again. Usually a minimum of six again. This part can go one of two ways depending on the tech who picks it up next and how many paragraphs you spend convincing them to actually look into the problem because the previous assumption was just that, and assumption:

1) They only read YOUR response, and don’t bother to do more than a quick glance at the entire ticket history and respond with more unhelpful nonsense based on the assumptions that they immediately make without looking into the actual problem or reading the original ticket and not just the response. (Sometimes, this step will have to be repeated to reach the next step.)

2) They do their job, read the entire ticket thread, look into the issue, tell you “oh yeah, you’re right.” and fix the problem.

It’s a crap shoot on how long it takes you to get to 4-2 there. More than half the time, it only takes one follow up explaining and explaining and explaining why the tech didn’t bother to look into the problem or they would have noticed ALL of these reasons why their assumption was entirely unhelpful and unrelated on their own, had they looked into it at all on their own.

Basically, what should be, at worst, a few hours for help, turns into well over a day of trying to convince tech support to actually look into the problem instead of trying to dismiss it as quickly as they feel they can get away with.

Our site is still down…

about fucking time

WHY do you have three NS … NS[1-3] … ya know, for redundancy. And they all go down at the same time? Are you kidding me??

This time we had no down time. This is good. Anyway I think you need a blessing…or maybe an exorcism, given the last problems.

I didn’t read this comment thread, but we’re still getting DNS errors from email on our email.

“Technical details of permanent failure:DNS Error: DNS server returned answer with no data”

I find this totally unacceptable. Our site has been down a few times already in only two months. We are leaving this server by next week, awful service and awfully slow support feedback.

@DelawareSmith I use Monitor.us

Like DH but the DNS is probably the single most SUCK in the service.

My DNS is SO MUCH BETTER THAN YOURS WILL EVER BE! But I need to drop too much $/domain to use my superior service. That dynamic DH IP is total bullshit and I know I get penalized in SEO because you recycle these IPS.

Keep up the good work, but ideally give me an option to pay $100/year to get static IP on my account (like my other servers).

@Brady Mills Thanks for the Monitor.us recommendation. I’m new to web design, one more year before I graduate with my BS from AAU SF and this kind of thing stresses me out. I had down time problems with my last host and just switched to DreamHost in January. Very disappointed.

I think Monitor.us will be very helpful to me and ease my stress levels. :)

Thanks again!

this makes me think that some of the problems i have been reporting with a ridiculously slow wordpress install (for several months, ongoing) is caused by this. the response i got from DH was that resources were being used by other domains on my account. now i think their answer was a bunch of malarky.

Will mails sent whilst the server was down make it to me?

Our client’s site is still down as of 5:22 Eastern time.

Hi! for me not resolved, my sites all are down! help please!

Now works OK, sorry

But two of my sites are still down! How can this be resolved?

So, what was the problem? I think that is kind of important info…

I’ll consider this situation RESOLVED when the internets knows how to find my server again. Until that point, I’m really hoping I don’t regret making the switch to Dreamhost from GoDaddy earlier this week.

My site is still down as of 5:17pm CST.

Over 90 minutes of continuous downtime as of 6:15 CST. Completely unacceptable to call this resolved.

Our site as well as our WordPress has been extremely slow all day. Not sure if this is related. All other sites I visit are fast like normal, so I’m assuming it’s server-side.

Well, according to the guy I just chatted up on the support site, they expect things to be “back to normal within one to two hours.” Funny that they are calling this resolved. FML for leaving GoDaddy two days ago.

4+ Hours of continuous downtime. Definitely not resolved.

Well, I’m in Georgia (USA) and all my dreamhost sites are down. :(

We’re now on hour 3 of the tech’s 1-2 hour estimate. Approaching 5 hours+ total downtime.

This sucks. big time.

This was the first outage (that I’ve noticed) since I moved my dns records to dreamhost a few months ago. I really appreciate the system status posts. Well done and thanks.

For those who’s sites are still not up, you’re likely waiting for your ISPs DNS servers to update their cache. They don’t all adhere to the TTL. You’ll probably wake up tomorrow to find it working. Here’s some info on DNS propagation: http://bit.ly/Ycf0Gt So while it’s fixed at dreamhost, you may be waiting for your ISPs dns servers to update with the fixed info that dreamhost is serving.

If you change your computer’s connection settings to use google’s public dns servers, you may be able to access your site (after you flush your local dns cache or reboot your computer.) instructions and ip addresses here: http://bit.ly/Ycfb4A

I have 3 of my sites still down. no website, no email, no webmail. Support ticket has been submitted since Dhost thinks the problem has been resolved.

Still experiencing EXTREME slowness on php pages of the site.

Still slow today. I have a site on MediaTemple. Shared hosting costs $20 a month, and I never have issues like this with them. So what is it with DH? I have been with DH for 10 years (yup!) and the sporadic problems never seem to disappear. Their hosting panel is the best, but the bottom line is that this kind of unreliability is becoming increasingly unacceptable.

@Roger Totally agree. Been with Dreamhost since 2001 and while there have been downtime and outages here and there in the past ten years, these past few months of slow site loading, dropped connections from the server, outages, etc. have been unacceptable.

Add to that the fact that one now has to wait 3-4 days for a response from support. I remember when they used to respond in 24 hours or less. I have to wonder why they are falling apart. I’ve already bought hosting from another company and am probably going to move everything over to my new host. I can’t effectively do business through online when my website takes 30-60 seconds to load, or doesn’t load at all.

So quick question… Can someone explain what happened? I’m obviously not happened that I suddenly stopped receiving e-mail. DNS of course, as everyone is well aware is the *most* essential service for any hosting provider. So can we get a good, forensic breakdown of what happened?

@Roger I’m going to look at them for all my sites, I need to look at options. This couldn’t have happened at a worse time.

http://www.dreamhost.com is down. my webmail is down. what is going on? This issue is definitely NOT resolved like it is claimed at the top of this page. What kind of plan compensation will we be receiving?

@Tibor Yes, it’s a shame they have gone downhill. I do like their web panel and their email newsletters but that does not make up for a crappy front-end experience which is of course what we are actually paying for. I have a ticket into MediaTemple to migrate 20 something websites, on several separate accounts I should mention.

@Justin You can expect no compensation, refund, etc. That’s another difference. When my email at MediaTemple went down, it was on and off sporadically for three or four days. Yes I was pissed, but I didn’t actually lose email, it was just delayed. And the website itself was fine all that time. And how did MediaTemple respond? By giving me three months free hosting! You see, that’s why DH is going to lose my business. If it wasn’t for the ordeal of moving sites, I’d have done it long ago. Now I see MT has a migration service, so I am just waiting for a quote on the cost.

I can’t believe I am spending time writing here instead of working. Oh wait, that’s right, I am waiting for one of my site pages to load!

@Roger Thanks for the info about MediaTemple. I’ve heard of them before, but have been with Dreamhost for a really long time. Typically, their service is decent. But this has been horrible. I’ll check out MediaTemple to see what their plans are and look into switching as well. I have about 20 or so sites that I’ll need to migrate over. So it will be a pain for sure, but it will help to know my options.

Having Email issues! Not with all of them but some!

What is going on with Dreamhost?

@Roger @Justin I will probably look into Media Temple as well. I also have about 20 sites that would have to be moved which is a huge headache but I guess I’d rather take the hit time-wise than keep suffering through this insanity. I just find it crazy that we are not being told what is going on, and that support emails are going unanswered.

When they are having outages like this it probably overwhelms their support staff/system. Still. It would be nice to have an idea of wtf is going on.

Anyone still having issues? My site’s gone daddy gone and I seem to have lost my NS. Is this what happened to you all?

I am still having issues. Some sites up and some not. Submitted a ticket this morning but…

My two hosted domains were having major issues this morning. I did the DNS refresh on both, and they started working again.

https://panel.dreamhost.com/index.cgi?tree=domain.manage -> click DNS under the problem domain -> under “Refresh DNS” click “GO!”

my all sites are down, jabber IM is down, FTPs are down. no one still able to fix it.

It is the late afternoon on Feb. 19th here in Japan.
My site is VERY slow off and on now for a few days.
A pro who is a friend had a look and wrote:

I took a quick look and deactivated a couple of the stats plugins but that didn’t seem to help any. You are getting a slow down and errors on the
backend and the front end. Sometimes pages don’t load at all, sometimes it
comes back with Server 500 errors. I think it is likely that your provider is having issues.

//
Any help or advice please?
These kind of issue really hurt my business.

Regards,
Charlie

Site completely down in Vancouver, Canada. Grrrrrr

When will DNS be up?

My websites on my servers are down!!!

I was having DNS issues where I was not receiving any email. The refresh DNS worked for me:

https://panel.dreamhost.com/index.cgi?tree=domain.manage -> click DNS under the problem domain -> under “Refresh DNS” click “GO!”

again my all websites are down MySQL database is broken

and no one is even attending chat.

Please wait for a site operator to respond.

You are not currently in a chat session.

You are not currently in a chat session.

All operators are currently assisting others. Thanks for your patience. An operator will be with you shortly.

All operators are currently assisting others. Thanks for your patience. An operator will be with you shortly.

There are currently no available agents to interact with you. Please try again later.

You are now chatting with ‘gary’

gary: We are having a network problem at the moment. The sys admins are working with the network operations center to resolve this. If it goes on very long we will post status at http://status.dreamhost.com . Very sorry for the inconvenience.

Chat session has been terminated by the site operator.

YOU SEE HOW PROFESSIONL WAS HE

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues with Web Panel and some services (RESOLVED)
11:55 am

Connectivity issues with Web Panel and some services (RESOLVED)

Posted (February 12th, 2013 at 11:55 am PST) by patrickm

We apologize for the issues you may be having with accessing your Customer Web Panel at this time, as well as any access issues with WebMail or your hosting services. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible. Thanks for your continued patience.

Update 12:20pm – This issue has now been resolved! This incident caused connectivity issues with your Customer Web Panel, WebMail and some hosting services. Our network engineers have identified the cause and have put a fix in place to prevent this in the future. All services are back online, but please contact Support with any questions you might have! -PM

This entry was posted on Tuesday, February 12th, 2013 at 11:55 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

46 Responses to “Connectivity issues with Web Panel and some services (RESOLVED)”

Many of my clients’ sites are down, and so is mine!
Hurry DreamHost, hurry! They are starting to scratch at the door!

My site is down and so is dreamhost.com :|

hahahaha. sorry but this is like the 5 time in as many days that my VPS has gone down. All I can really do is laugh.

It’s 3pm Eastern – still down. What’s the deal? DH staff you need to update your customers.

So is all my stuff. :-(

they’re trying very hard to update their customers on time. what they should work for is better up times. this is getting ridiculous.

The last 2 weeks have been a lot of little outages, but this is the first one where all of Dreamhost stopped answering. Recently there was a DDOS attack on the server my sites are on, this is starting to look like a coordinated assault on all of Dreamhost, something needs to be done.

my vps is down as well.

Patience, patience, we’ve had six power failures here in the last few weeks, answer from the power company? Nothing…some bird flies into a wire…or a switch, or trips some arm on a transformer…that’s the ANSWER?!!

At least here you can instantly see what’s happening, and you HAVE a status page to visit so you don’t go off the deep-end.

Keep at ‘er DH Happy connectivity resolving crew.

Well, this has been recurring for a while, I was happy I had been moved to a new server, but obviously this is the dreamhost standard, I just had one client asked to be moved. Thankfully others have not noticed.

I had high school students waiting for their assignments. It was down for about 5 minutes, right at the beginning of class. We’re back up now. Thanks for the quick fix DH.

What ever server the “dreamhoststatus.com” site is on, I’d like my sites moved to that one, it seems to always be going no matter what…

The fact that it took this long for a blog post to be published is absurd. I reported the issue at noon (still no reply)! My websites have been down for nearly 6 hours this month, your killing my business!

I have been a customer for years, the last few months have been like watching a slow motion train wreck. All of the outages and the total lack of support are unreal. I feel like a fool for recommending you so often and like an idiot for staying with you so long.

Your on thin ice.

Second in the last few days…what’s going on?

Alicia and others — times posted are PST, not EST. your comment was just a few minutes after their update, so hang tight!

I am up now. Down time wasnt too awful. Thank you, DH.

I’m not talking about the time it took for me to reply to the blog post. My sites went down at 11a EST, I sent in a ticket at 12p EST and the wizards at dreamhost did not post this till 3p EST. If they did not notice an outage of this magnitude for 3 hours they need to seriously consider becoming a reseller for a more reliable hosting company.

This seems to be getting very common – almost weekly since I joined Dreamhost late last year. My site is still very slow/crashing now – any further updates yet?

My cached pages seem to be OK, it’s the none cached pages that are dying which suggests it could be a DB issue … yeah?

I’m right there with you, Mike. I’ve been a happy DH customer since 2006, many of my clients use DH based on my recommendation, but the last two months have been really frustrating – something DH needs to address directly with its customers, not just with these status updates.

For those of you not running major businesses, then these ‘little outages’ may not affect you to the same degree so bear that in mind when you’re advocating patience, please.

Hi, our site is still down although DH is saying everything is back up. Can someone please follow up?

@Mike: Your outage seems to have been unrelated to the issue reported in this post. Only widespread issue affecting lots of customers will be reported on this page. Support will be in touch with you on your issue, or if you can give me some indication of which domain is having problems, I can check it out more directly.

My site and my husband’s is still down! (bonnielangfitness.com and survivethrive.org)… Will this be resolved soon?

Hi Patrick – all domains were down. Those using SQL and not. They are back up now, still nothing from support.

Is there a premium support package I can buy that garners a reasonable response time? 24 hours does not cut it. I need a response in 2-3 hours tops.

First 7 years with dreamhost…. great!!! …. last 2 years with dreamhost…. WTFFFFFFFFFF !!!!!

I’ve been having SQL issues since at least Saturday night. Outages since then for anywhere from 10 minutes to 1.25 hrs and seemingly random intervals.

Support has told me a few times now that they don’t see an issue (of course when they check – there is no issue), now my issue has been kicked over to a MySQL admin.

Server is mudskipper. Is this networking issue possibly causing my issue as well.

Currently it’s been down for 13 minutes.

It is not resolved. Sites are still down.

I know this says resolved, but my sites just went down. been down for twenty minutes now.

My sites are also down, have been since yesterday.

Please update this issue so that we know what’s going on; it’s clearly not completely resolved.

WebMail and Web Panel still unreachable.

@Brian
I have also been having issues with my SQL VPS since this weekend and of course when DM checks, everything is fine. But, I log the response times for connecting to the server with Pingdom, and I am clearly seeing a spike of a 10x increase in response time since this weekend. It is still occurring even today.

Our email is still not available, since yesterday morning ( Feb 11). We run our business through this site, so we are not very impressed with this ongoing outage. When will it be back up?

It’s still down for me too. I can’t access the panel or FTP. Ouch!

Yes, still down for me too. My sites are down, I can’t get to the panel, can’t ssh in — nothing.

It was nice of Dreamhost to charge me the yearly fee the day before yesterday. But my mail and site is down today. Please fix the problem.

I’m an idiot please disregard my previous comment

I’m really frustrated with this, can’t get through to support, can’t get anything to work. Is DH still in business? Are they expecting to stay in business? Should I be looking for a new ISP?

@James
2 things for me –
I was able to move my database to another server. I’ve always had databases on two different servers, so I chose the option last night to merge them to the one server.

That combined with some looking into by Justin @ DH – he modified a couple table to improve performance on a couple queries I’ve been running.

Not sure which one fixed the issue, but as of this morning – not a blip. Been working awesome since.

This is pathetic… 2 days and some of our client sites are still down!

According to Pingdom, our site has been down more than 12 times in the last 24 hours. We are on the liquorcabinet server. It’s so frustrating.

Our MySQL server is down now

We have two sites down, one with a 500 internal server error and the other ‘server not found’. Can’t get to mysql either.

DH that’s the kicker. moving off from you guys by end of Q2 this year. enough is enough.

Seems our sites are down…I guess the issue/problem was never fully solved.

was down on the 12th, down again. Been down for about 20 mins now.

I can’t enter into the Dreamhost Panel. And most my webpages are down.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Service connectivity issues for 2 racks of servers in our US-West Los Angeles datacenter (RESOLVED)
12:25 pm

Service connectivity issues for 2 racks of servers in our US-West Los Angeles datacenter (RESOLVED)

Posted (February 9th, 2013 at 12:25 pm PST) by patrickm

We are currently experiencing connectivity issues on 2 racks of servers in our Los Angeles datacenter. This is affecting all web services for the servers listed in the first comment of this status post. Our engineers are actively working to resolve this outage and we will be updating this post as we have new information.

Please contact Support from the Support > Contact Support section of your Web Panel if you have questions.

Update February 9th 1:38pm: Our network engineering team has given the all clear and we are no longer seeing any packet loss. Should you have any further issues with your sites being down, please contact our support team via the web panel by going to Support -> Contact Support. -Oscar

Update February 9th 1:25pm: We are no longer seeing packet loss for these two racks, but are continuing to monitor traffic to ensure there are no further issues. -Oscar

Update February 9th 1:05pm: Our network engineering team is still investigating the root cause of this issue. We are seeing intermittent packet loss for those two racks. Will will continue to update this post as we have more information and we appreciate your patience while we resolve this issue. -Oscar

Update February 9th 12:45pm: Services are reporting restored and our tests are successful as well. We are still monitoring the situation to make sure there are no further or lingering problems.

This entry was posted on Saturday, February 9th, 2013 at 12:25 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

26 Responses to “Service connectivity issues for 2 racks of servers in our US-West Los Angeles datacenter (RESOLVED)”

Affected servers:

homie-vserver23 wolverine homie-vserver53 homie-vserver18 homie-vserver22 homie-vserver56 homie-vserver25 homie-vserver29 homie-vserver16 homie-vserver31 homie-vserver32 homie-vserver42 cyclops catwoman castellano pistone buchalter walsh terranova mcgurn gotti deadpool homie-vserver34 gravano nitti robin venom kakuji zwerbach greene spillane cummiskey brasco legs capone homie-vserver230 fitzpatrick bulger coonan eastman mclean colossus

For the homie-vservers, please see the Account Status dropdown in your Web Panel to see if your VPS is affected.

I can confirm the downtime and slowness, but you’re quick in resolving this! Thanks!

February 9th, 2013 at 1:19 pmScott Cranfill Says:

I’m experiencing issues with my hosting on server “coffee”, which is not in the list of affected servers.

what was the problem?

Thanks for being on top of it!

It was fun to run a tracert for the first time in 10 years anyhow…

thanks for resolving this qo quickly, site was super slow

Doesn’t appear to have been fixed for VPS…
~4:40 and we’re offline

all domains on that vps are down…

going into account and rebooting the vps – thought that might have been something that would have been checked….

Are there any similar packet loss issues for ‘homie-vsql20’?

My server is on this list, homie-vserver18, and it is having trouble again!

Extremely slow response loading pages.

I believe our server is ‘bath’.

on accomack server and about 4-5 hours ago my site is barely responsive

Our site is down and I can’t access the webpanel to submit support ticket.

Same here, everything is down again. This is crazy.

everything dreamhost seems unreachable right now.

including http://www.dreamhost.com
and of course my site!!

Our site is down :(

Panel is unavailable .

All sites down. Can’t get to deamhost.com.

How long til we get an updated status report? All sites in development are inaccessible at this time. Deadlines don’t wait…please update immediately on status.

Wow, they won’t even give an acknowledgement after everyone being down for >20 minutes, great communication there

Website, FTP, and C-Panel are down.

I try emailing support and it says to fill out the form on the site (that is down) makes a lot of sense :(

Escobar is down. Can confirm FTP and panel are down here too.

Everything 100% unreachable including dreamhost.com

all other websites(not to do with dreamhost) are OK.

Sporadic poor performance for the last 16 hours.

It’s back up, thanks!

Seems ok now.

It is extremely natural to be able to desire, however these additional calories may easily develop into stubborn tummy body fat. This is probable on the comfort connected with own cooking area. However is their own medical insurance securely within location?

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Issues Between our Downtown Los Angeles and Irvine Datacenters (RESOLVED)
7:02 am

Network Issues Between our Downtown Los Angeles and Irvine Datacenters (RESOLVED)

Posted (February 7th, 2013 at 7:02 am PST) by Justin@DH

We are experiencing packet loss in traffic between our Downtown Los Angeles and Irvine datacenters at this time. This is causing specific issues with customers with split webserver and mysql services between the two datacenters, and mysql servers in the irvine datacenter. Our network engineering team is online and looking into the issue currently. When we have further information we will update this post.

Update February 7th 7:50am PST: The issues appeared to be gone for a short time but have returned, we have all our admin and network engineering team on this issue at this time. Our apologies for the issues this is causing those customers affected.

Update February 7th 8:20am PST: Our network engineering team has been able to fix the issue, which appears to be related to a DDoS from specific sources. These have been corrected and further investigation for hardening of the network will continue. Our apologies for the issues you experienced this morning.

This entry was posted on Thursday, February 7th, 2013 at 7:02 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

32 Responses to “Network Issues Between our Downtown Los Angeles and Irvine Datacenters (RESOLVED)”

Which servers are affected? Tuba & Burnet, possibly?

@Craig: Yes, tuba is affected. The most affected are cornet, euphonium, gong, tuba and whistle.

I am on VPS homie-vserver56 and all my sites are down. Error establishing a database connection. Also, in the Panel, all of my databases are showing 0KB in size. Really worrying me.

my site is down.. how do i know if this is the cause of the issue or not?

not slight at all – 100% down – mysql servers not responding – this is why you should put my mysql server on same net segment as http server.

again dreamhost fails me.

@Brian: It means nothing what the panel says for database size in Goodies -> MySQL Databases. That value only updates weekly, and if you were recently mysql moved it would show 0kb. You are affected by this issue.

@jon: There isn’t enough information in your comment to tell if you are. This issue only affects a small amount of customers.

@kj: Most customers are all in the same datacenter, but there are some older accounts that may be affected as they were setup before more control on datacenter location was established. We can move you to the same datacenter when the issue is resolved, if you submit a ticket.

is this the problem affecting my websites? (spearmintbaby.com)

this is the message I am getting when I try to log in: The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

My servers are in the east. Sites are up but can’t ftp to any of them due to “network latency” issues I was told.

“slight packet loss.” Sounds like an oxymoron to me.

I flagged this as an issue over 24 hours ago, it makes me pretty sad that it has only been raised as an issue here 30 mins ago. :(

Our site is down. We are in Louisville, KY. Please advise when back up! Thanks

It have been 6 hours now… None of my wordpress sites works. I have Ad campaigns running in some of them.
Please hurry up!

I have two client sites down (two separate DreamHost accounts) and both are down. One site just had it’s “grand opening” on Monday so this is NOT a good time to be down.

Hope you can resolve the issues promptly!

@GB: if you are using Adwords, you should be able to disable the campaign temporarily to prevent any loss. Just a thought. -And I am in no way affiliated with DH. Just saw your comment and thought it might be worth a try.

@Rene Michaels: Just tell them that the site is getting pounded with traffic that wasn’t expected.

@GB same here and status message was published after hours searching and support tickets without answer, hope this ends soon.

OMG… what’s happening??

@Brian

Thanks, but I don’t lie to my clients. Having said that, this does look like a DOS attack doesn’t it? I got a spam comment this morning and when I tried to login to moderate, that’s when discovered the site was down.

I’ve already told my clients I am monitoring the situation and will let them know when resolved.

To those of you who are down on DreamHost, this happens from time to time even on the “best” hosts. I have other accounts with other hosts and have experienced similar downtime over the years. It’s the nature of the Internet! We can all pray that this will be resolved quickly.

My email (homiemail-sub5) is very slow this morning – is this a recurrence of the problems with that server, or related to this LA/Irvine issue? Anyone else seeing mail issues?

FYI: I CAN access my directories via FTP client just fine.

We’re experiencing it on our sites too. It’s hard to submit a ticket when I can’t get Panel.dreamhost.com to come up too. :) Know you’ll get it fixed but soon would be nice.

@Rene MIchaels: I was just joking about that. I actually do however get to say that to my clients now and then. Some of my sites push through extremely high amounts of data with software updates for my company. 1 update with our software will equate to 50ish MB files being downloaded from about 2000-3000 customers in 1 day. We do some offsite cashing and mirrors but it still can slow things down.

YAY!!! back up. lets hope it holds.

Back again, hope this last for a long time. Thanks DH

Still no MySQL on my website.

@KB

I was able to access my control panel but it appears to be down again. I am on US – West (Los Angeles, CA) for email and web control panel, and on US – West (Irvine, CA) for web services.

Working now. Looks like they swapped out for a fresh flux capacitor.

is poplar server affected?

Do yall know if mail is being affected, or delayed?

Example I got 6 emails before 9 am est and none since then. Since I got over 75 time sensitive emails on that acct yesterday I should have more emails right now.

@Diego:

If you can access your control panel, login and click on “Status” then “Data Centers” and it will show you what servers you are on.

We need to know when our site will be available, this is a must for us.
Regards.

My site is down in Chicago as well….

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Email Downtime for Homiemail-sub5 (RESOLVED)
5:16 pm

Email Downtime for Homiemail-sub5 (RESOLVED)

Posted (January 31st, 2013 at 5:16 pm PST) by Dan J

We are currently experiencing a few problems with our homiemail-sub5 mail server, which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Jan. 31st 5:58pm PST: The file servers are back online now. Our admins report that the mail appears to be operating properly now and there should be no delay for any mail at this time. We are continuing the investigation to ensure this will not happen again but for now we are resolving this post. If you do continue to see any problems then please contact support through the DreamHost panel.

Update Jan. 31st 5:39pm PST: We are still working on the issues with the homiemail-sub5 server. We have admins onsite and expect to have everything back online shortly. Again we apologize for any inconvenience that this is causing you.

This entry was posted on Thursday, January 31st, 2013 at 5:16 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Email Downtime for Homiemail-sub5 (RESOLVED)”

Again? Like the third time in a couple weeks this server has been down.

That means that there is a problem and that you should fix it.

homiemail-sub4 looks like halted too.

This appears to be only affecting homiemail-sub5, though if you’re having problems with homiemail-sub4 email addresses then please contact support through the Dreamhost panel so we can look into that for you.

I contacted support and they said announce lists are in trouble too.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)
4:51 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)

Posted (January 31st, 2013 at 4:51 pm PST) by Ralph

We are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May is finally nearing completion. We started this project when we realized our network stability was in danger as we had ourgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, February 6th we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail, MySQL and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update February 7th 3:40am PST: Everything is back to normal on the old setup. We will reschedule this maintenance and post to let you know when. Thank you for your patience. We are confident we learned from tonights events in order to have a successful maintenance on our next attempt.

Update February 7th 3:25am PST: Revert is complete. We are just watching to make sure everything returns completely back to normal.

Update February 7th 2:45am PST: We are opening this post up again as we have decided it is best to completely roll back this maintenance. We were noticing stability issues that we were not comfortable with, even though the issues were resolved quickly. Roll back is commencing now.

Update February 7th 1:15am PST: We are concluding tonights maintenance. We got most, but not all networks successfully moved to the new hardware. We will need to revisit this maintenance in a different announcement. Good news is that we did complete a significant portion of this maintenance. Thank you again for your patience and the extra time used to extend the maintenance.

Update February 7th 1:10am PST: We are having issues troubleshooting this network and during the troubleshooting some mail service issues occurred. We limited service impact to less than 10 minutes. We are making some final moves to the new hardware before packing up. Thank you for your patience through this maintenance.

Update February 7th 12:30am PST: We continue to troubleshoot the previous network we attempted to move. We have it running on the old hardware still. Once again, we have extended this maintenance to at least 1am Feb. 7th since we have been able to minimize service impact.

Update February 6th 11:45pm PST: Everything was proceeding smoothly until we got to our most recent network. We rolled back that specific network to the old hardware while we iron out the issues. We will update as soon as we successfully complete this part of the update. As of now we are likely extending this maintenance at least 1 hour.

Update February 6th 11:20pm PST: We made it successfully through the vlan that was giving us trouble. We were able to keep service up during these difficulties. We are now moving through other virtual networks without incident. So the maintenance is back to chugging along again.

Update February 6th 10:40pm PST: We are having some issues with the current vlan we are moving. We were able to roll it back to the old hardware to limit downtime. We have since completed the move to the new hardware and we are ironing out some final issues. There should be no active issues right now, and even with this hiccup we were able to act quickly to minimize service impact.

Update February 6th 10:05pm PST: So far the subset moves are going smoothly. We are seeing the moves take a matter of seconds. Our first runs were on lesser used networks and we are about to get to larger customer populated networks. We will be watching the comments, support and the move itself closely to make sure everything continues as planned.

Update February 6th 9:40pm PST: That went well, so now we’re starting on moving small subsets of networks over to the new network hardware. There should only be a second or two of downtime during the switch. We’re monitoring this to ensure these changes do not escalate into a longer issues.

Update February 6th 9:15pm PST: We’re now updating some of our traffic routing. This one should have no customer impact, if things go correctly. We’re watching this one closely.

Update February 6th 9:00pm PST: Part 1 is now complete, we’re moving on to the next phase. We’re testing now, to ensure we’re ready to move on to part 2!

Update February 6th 8:45pm PST: We’re now taking our first step here, we do not expect any customer facing downtime.

Update February 6th 8:30pm PST: As of now we’re a bit delayed, but are gearing up to start things momentarily. Our first step is to move traffic from our old network hardware to the new hardware. We’ll update, when that happens.

Update February 6th 7:58pm PST: We’re getting ready to start the final phase of our series of network improvements. We’re excited to get the ball rolling!

This entry was posted on Thursday, January 31st, 2013 at 4:51 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

34 Responses to “Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)”

Here’s wishing you all lots of good luck and may Murphy stay the hell away from everything tonight ;-)

Didn’t realize about the move, tried updating my website, major problems with updates, even was kicked out of the backend, sent to error message pages. I am glad I checked here to see what may be going on. Hopefully all will be normal soon:)

@dee, you shouldn’t see any downtime at the moment, so, if your site is still broken, please let us know your domain, and we’ll check it out for you.

any chance we will lose files we are working on at the moment?

@M, depends on how exactly you’re working on them. Its best to work on files on your computer, and upload copies to our server, so you don’t get caught by any issues. These are only network issues, however always best to be safe.

My DB VPS is down. Basic HTML sites load fine but DB driven sites are showing DB errors.

@brian, Already do, using FTP client. I for some reason thought these were server upgrades as well. Thanks for the timely reply.

Hi Brian,

Just checked, I am able to log in to the back end and make changes, although at times it is a little slow. I am glad that I wasn’t hacked, like I originally thought may have happened.

@jen, we’re looking into it for you!

@jen, the mysql VPS server you’re on is having problems. Our support staff will be sending you out an email shortly!

My websites have been down sporadically throughout this evening. Font.My, Fontogram.com, and Font2Go.co … Do you think it’s related to this maintenance?

Looks like I am getting timeouts connecting to my domain – web, ssh. Might have something to do with this?

@Pat, good question! We’re investigating that. I can’t get to the IPs your domains are on, so, its very likely

@cedric, you listed http://cedricandcharlotte.com/ as your domain, but, that domain isn’t registered. Is it spelled correctly, or do you have another domain on our servers?

Experiecing some problems accessing our website and ftp and we’re sitting here with our developers trying to update. I would appriciate an update on when this would be resolved.
ping times out…
Thanks.

@made, we rolled back our latest change, so it should be working ok for now.

Wuhuuu…we’re back up.

@Brian, Please help resolve this as soon as possible. Thank you so much! Our App needs to access the server to process information, and customers are bombarding me with questions!

Yay. Works now. Thank you!

sorry I got the order wrong – it’s charlotteandcedric.com!

seems to be working again btw.

I like the old hardware a lot better….can we just keep that :-)

Murphys Law
Everytime I get the motivation to update/finish websites, it’s maintenance.

WHY U NO WRK ? hurry up :P

Still on and off. Restore the stability as soon as possible please.

-As a Network Architect/Administrator
I feel your pain guys, also, maintenance nights = best laxative ever.

-As a Customer
Hurry up ! ;)

Here in Europe: maintenance days :(

Here in Europe: maintenance days = worst flustrating ever

The maintenance is currently in troubleshooting mode and all service is working correctly. I have tested all your sites, @Robert and @Pat and they are working correctly. We are able to troubleshoot without keeping your services down, which is great. We hope to have this maintenance finished soon. Sorry about the European time different @Robert.

Up and working, thanks :)

Our site is on again and off again! We are trying to send files to customers and its getting frustrating!

@M we will be stopping the maintenance shortly. We did not fully complete the maintenance yet we want to stay as close to true on our window as possible. We are basically cleaning up and will revisit any final networks to move at a later date.

my site creudance.com appears down

@jason, looks like your apache server was hung up, unrelated to the maintenance. I fixed that, so your site is now up!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) Part 1/2 of Final Step – Wednesday January 30, 6pm (4 hours) (RESOLVED)
4:01 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) Part 1/2 of Final Step – Wednesday January 30, 6pm (4 hours) (RESOLVED)

Posted (January 25th, 2013 at 4:01 pm PST) by Ralph

We are nearing the end of the Network Improvement Series that we started back in May of 2012. We are very pleased with the performance of the new hardware we have installed to replace our previous network set up. Wednesday’s maintenance, on January 30th, in our US-West Data Center (Los Angeles, CA) is not customer-facing, however, changes made in the DreamHost panel (modifying or creating domains or users, for instance) may be delayed. We are scheduling a wide four-hour window to complete the maintenance, pre and post testing, and to cover any unforeseen complications.

You can check if you’re potentially affected by this on the DreamHost panel.
If you have any questions about this before or during the process, please let us know by contacting Support. We will be happy to help. During the maintenance, we will also leave the comments open and will be monitoring them as we provide periodic updates during the maintenance window.

We apologize for any inconvenience this may cause and appreciate your continued patience while we further improve our infrastructure.

Update January 30th 9:20pm PST: We’re all set, this step of our maintenance plan has been executed and completed. Thanks to everyone that helped and provided feedback!

Update January 30th 9:00pm PST: We’re winding down here and tying up loose ends. We did see some issues with a single server, and that has now been resolved.

Update January 30th 8:19pm PST: We’re still going strong, we did run into an issue with domain registrations. Any new registrations may have failed within the last hour. We apologize for this, and it should be fixed now. We’re monitoring this to make sure it stays working.

Update January 30th 7:34pm PST: The maintenance window is still happening, we had a small issue with our signup page, but that has now been resolved. Truckin’ right along!

Update January 30th 7:00pm PST: Still plugging along here, all on our backend, so no one should be seeing problems related to this.

Update January 30th 6:32pm PST: So far, we’re doing fine, no customer facing issues at this time. We’ve updated our VPN routes, and are now testing those. Again, you shouldn’t see any problems related to this, at this time.

Update January 30th 5:58pm PST: Our team is in place and ready to start this maintenance. Comments are now open if you feel you are having issues. This maintenance is not expected to cause any service problems, but we are here to help just in case. We will update as we progress through this maintenance.

This entry was posted on Friday, January 25th, 2013 at 4:01 pm and is filed under General Outages, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

22 Responses to “Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) Part 1/2 of Final Step – Wednesday January 30, 6pm (4 hours) (RESOLVED)”

Is anyone else having trouble with ‘signup.dreamhost.com’ right now? Odd that it’s unresponsive (after you enter a domain name) *just* now.

My sites are down since morning and nothing improved since. How long it will take to restore the functionality back?

@Lars K – Signup is working correctly now. @Lizy – your issue is unrelated but we have a tech working on this right now!

@Lars, we had a small issue with signup, however it should be fixed now!

@Lizy, I’m investigating that, I’ll have more info shortly!

@Lizy, you listed https://scribbletone.com but don’t have https set up. http://scribbletone.com works, did you have https set up in the past? It requires a unique IP, so you’d be paying for that as well.

@Roman, I tested the site you listed, http://romansima.com and it seems to be working, are you still seeing it as down? If so, please let us know!

@Roman – site you linked is working for me. Site outages are not related to this maintenance. I checked your account and all domains are working correctly for me. If you are currently having problems please run a traceroute, wiki.dreamhost.com/traceroute, and provide that in a support ticket.

Possibly a coincidence but my sites just disappeared about 20 minutes ago.

@Russ, I just tested the site, and it’s working for me, are you still seeing it down?

@Brian Yes, it still doesn’t work on my end. Actually none of them are working. Can you check the following: http://www.romansima.com | http://www.romansima.me | http://www.romansima.com/me | http://www.romansima.com/blog | http://www.concept-guru.com. All are separate installations. Why they are not working?

@Ralph I understand, but it’s going since morning like this. I’m on EST. Also, I tried your suggestion and it gave “-bash: syntax error near unexpected token `newline'”. What now?

@Ralph, @Brian,

Thanks for following up on that! Signup works again, and I got myself a referral. :)

@roman, have you submitted a support ticket? I don’t see one in your support history, so please submit one now, and a tech will check into it for you. This isn’t related to our maintenance, and since it’s working for us, needs to be looked into by our support staff.

@Brian OK, noted.

Can’t reach it from home or phone and isup.me says its down.
I’m going to bed – fix it before I wake up. ;-)
It disappeared just before 9pm CST in the middle of a blog post.

Too little too late, and too much downtime, web hosting is seriously not rocket science. 90 days of pure hell being a DH customer and glad to be leaving. It is as if DH lost the only person who knows how to run a data center, and the rest of you are standing around watching each other wondering what to do next. Absolutely no way any customer should have 40 PAGES of support history after only 3 months…and now in the final throws of my migration away from DH, the cPanel goes down…BRILLIANT!!

And how pissed I am for all the times support tried to point the finger at my website templates, or some plugin that has been running for 2 years on another host with no problems, now I just tell them to reset Apache and they are like…ok…no questions asked cause they are so use to doing it.

To put a fine point on it…DreamHost sucks.

I have not been able to register a new domain for over an hour here… was working fine right before this maintenance started… Right now there is no response from the Web Panel when trying to register new domains… frustrating!

@joe, Our Tech Support manager is evaluating your support history now. We apologize for any issues you had with us. It does look like you had quite a storied support history with us. We’ll email you directly with more findings on where we went wrong for you.

@Duane, ouch, does look like that’s an issue. We’re investigating that! Can you test again and see if it works now?

THANK YOU!!!! Fixed (and I was able to register the domain I wanted)! I wish I would have posted here about 2 hours ago!! Thanks

@Duane, yeah, it’s nice to see DreamHost is now paying attention to customer issues in the place where many customers are likely to go when there *is* an issue. :)

Looks like we have a single server that’s affected by this, monarchs.dreamhost.com. @Russ, this is the machine you’re on, so we’re working on it!

Monarchs.dreamhost.com should be good now!

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail downtime on homiemail-sub5 (RESOLVED)
5:34 am

Mail downtime on homiemail-sub5 (RESOLVED)

Posted (January 25th, 2013 at 5:34 am PST) by chalise

We are currently experiencing a few problems with our homiemail-sub5 mail server, which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Jan 25th 7:35am: The switch has been replaced and all necessary configurations have been applied. Mail services for homiemail-sub5 are now online and are processing mail. We will continue to monitor this situation to ensure all services are working as expected. There is a delay in receiving email as the mail queues are processing incoming mail again, but should clear up within a few hours. We apologize for any inconvenience this caused and appreciate your patience while we resolved this outage. -Oscar

Update Jan 25th 7:15am: Our network engineer team is still working on replacing the affected switch. We will continue to update this post as we get more information. -Oscar

Update Jan 25th 6:45am: Our team is currently working on replacing the affected switch. We’ll continue posting updates until this is resolved. Thanks again for your patience!

Update Jan 25th 6:20am: Our network teams have identified the issue. It looks like our power strip’s internal breaker tripped, which has caused issues with the switch providing networking to the rack that the homiemail-sub5 filer is on. Our teams are currently working non-stop toward a resolution. More updates to come!

Update Jan 25th 6:00am: Our teams are hard at work looking into the source of this outage. Thank you for your continued patience and, again, we do apologize for the inconvenience this is causing! We will continue posting updates until this is resolved.

This entry was posted on Friday, January 25th, 2013 at 5:34 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

118 Responses to “Mail downtime on homiemail-sub5 (RESOLVED)”

how will we be notified? When will we be notified that it is fixed?

So what are the problems with this server? It seems that every now and then it gets shut down.

Pleeeeeeeeease fix it.

Emilio, Dreamhost is usually pretty good about updating these blog posts. Just refresh this page periodically. Or, clearly, when your email comes through. :)

Now I realise why I do not receive mails last couple of hours … Hmmmm
Hope that will be solved soon.

yes, our emails don’t work!!. We hope you could resolve it soon. thanks!!

u are constant having a problem when will this be fixed
thanks
kathy

homiemail-sub5 is our karma. again and again

January 25th, 2013 at 5:45 amMicrocadSoftware Says:

Our mails fails to.
Please try to fix it as soon as possible.
Thank you very much for your work and help¡¡

Sent in a ticket a few minutes before this was posted, so I’d better go and delete it to lighten the queue.

Thanks for the heads-up!

I’d love to check to see if I’m affected through the panel…but my primary e-mail is hosted with you, and I have enhanced security enabled. Whoops!

Pyro – to check your server log into the DH web panel, which is independent of email, and click the “account status” link in the upper right corner. Near the bottom of the pop-up window your email server will be listed.

I’m also experienceing problems on homiemail-sub4

This is the 2nd time in a couple of days…. not good….

Hi,have any notice about the server?

Our mails fails

Same problems on homiemail-sub4
Please look into this also!

Hello
all the people in our company can’t send or received any email.
please your help as soon as possible…

we are in the middle of something and it is REALLY IMPORTANT…

Thanks
Yohan

Please fix asap, I can’t afford any more downtime with clients, they are getting furious!

Sorry but really disappointed!

Every week we start to face different problems! Our business has been really affected!

is this a joke? this happen every week is seems. not a good way to start the work day. fix ASAP!

Where is this “Account Status” link that you speak of? The closest I see is “System Status” which links to this page. Judging from the webmail login failures, I must be affected. :S

any update?

not like january 2012 ? (remember?)

All e-mails from all sites hosted with you down! This is crazy! They want to kill me? This can not happen! Here in Brazil it is morning day and all companys need access to their emails! Help me!!!!!

jan2012 was a f*cking nightmare from hell

fix this shit

I am also affected and waiting for resolution.

Updates would be nice guys.

all email accounts down…

miiiinnccchiaaaaaaaaaaa!

update?

People are dying… hurry up, please.

and what if our important emails are missed (not received)?

All my emails are down… can’t even send out an email to clients telling them that…

One of our clients who emailed my gmail address report they are getting kickbacks to our email. Awesome!

I trust in the professionalism of lords, resolved ASAP please!!

Seriously, I have a $2m bid due this afternoon. All my sub-contractors e-mail their info, and my bid is due electronically.

FIX THIS ASAP PLEASE!!!!

Calm Down everybody… Dreamhost team surely is working har to solve this asap!

Can we have an update please???

when will this be fixed?

Seems like we had a very similar issue on this mail server last week:
http://www.dreamhoststatus.com/2013/01/17/mail-downtime-with-homiemail-sub5-and-webmail/

Hope it can be resolved asap as critical to business.

If your clients get kickbacks, that would let them know something is wrong, so that’s probably a good thing.

wait, that is a joke right?

“our power strip’s internal break tripped”

What utter shat!

Patience… Our E-mails are also extremely critical for our business. Thankfully Its been a long time since the Dreamhost E-mail has been down.(At least for us) We also have backup e-mail provisions just in case.

Issues lately are occurring too often. I’m not pleased at all.

This complete shutdown comes after a week of severe delays concerning email which you attributed to some other problem… Perhaps.
In any event, why do these problems and shutdowns happen so often with you guys?
When is it going to be fixed? C’mon, enough already.

It’s amazing how vunerable one feels when a simple thing like email stops functioning. I too am waiting for some very important emails. Makes one think of setting up different routes.It would be nice if D H would give us an estimate on the length of the outage :(

Our website is also not responding: http://www.davidkirktreehouse.com/index.php
Is this due to the same issue?

@Pierre – gmail is hosted… by Google… so if they’re receiving bounces it’s due to some other problem. The bounce message from the Gmail mailserver *will* have a plaintext bounce error code and explanation in it, so ask them to read it out to you (or forward to another email address).

And once more, to all those who SOLELY rely upon a single provider for critical services like email: read up on “single point of failure”. Compare with your setup. Make changes. I have active email accounts and email hosting with at least three providers, plus my fallback GMail *and* inactive accounts on a shared hosting backup service. Why? Because that’s what you need to have.

Every time something like this happens you get the same nickel-and-dime ‘business’ customers complaining about their loss of service ‘for their customers’ or ‘clients’. Dreamhost shared hosting is NOT business grade service! And to be fair, until this point I’ve not had a serious outage of any kind on my DH hosting for about a year now; this time period includes a cluster move AND this month’s data centre migration).

Dreamhost is good, but it’s still bargain basement style pile-it-high hosting. YGWYPF. A lot of people’s expectations seem curiously out of touch with reality.

If you’re using any best-effort shared service like Dreamhost to host business clients, your business model is fundamentally flawed. Get a VPS or dedicated server – from Dreamhost or anyone else – for your clients; segregate them from the unwashed public (i.e., all of us with DH accounts). I do exactly this for my own webhosting clients, I wouldn’t ever contemplate hosting business clients on a shared hosting service where anything like this can happen with no availability or time-to-fix SLA, it’s ultimately your reputation on the line and your fault for being a cheapskate.

Whinging about how a shared email server has failed garners no sympathy from the rest of us regular customers with reasonable and realistic expectations – and who are fed up with seeing your puerile, repetitive, shouty comments.

Kudos to DH for beginning the fix, considering they’re GMT-8 some technician’s phone will have been ringing at 5am to alert them of this so you can’t fault them for at least logging the fault and getting fault resolution underway. Their level of customer and tech support far surpasses other more expensive hosting companies I’ve used in the past. Be patient, and use this temporary downtime to think about how you can mitigate this kind of situation should it happen again in future. If you can’t afford for it to happen again, you can certainly afford to spend more than $100 on hosting.

1 hour problem, this is crazy!!!

Perhaps you have of the rats that eat cables? that is why you do not resolve the problems permanently! BOUGHT A CAT!

I’m packing my bags…

How long to resolve????????

1 hour and a half down, friday mornig uff!. Get a Cat!

This server is also showing on block lists now. I am unable to send emails to domains such as hotmail, live, msn etc.

Please unblock yourselves as spammers urgently we need to be able to send email

oh and please fix the problem as well. which is independent of the spam list you are now on !!

How long does it take to find the solution?

Yeah. What Christopher Woods said.

i’m not be patient at all, this sucks!

4 companies are stalled because of this problem!!

Making it hard to call this Finish Stronger Friday!

Please your urgent help solving the problem because the servers are not working and backup servers?

This is really not acceptable. We DEPEND on immediate communication with clients and this is a huge problem on a Friday morning. Time for a new host, maybe.

When will it be back up????????

Mr Woods, I would agree with you, except one thing… DH touts on their website very clearly that they have 100% uptime guarantee, so open up your tickets cause based on that they should issue you a refund on your next bill, 1 day per every hour of downtime..

Here is the full text from the DH website
::::::::::::::::::::::::::::::::

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

Add my name to the complaints list.

I have enjoyed DH for personal and casual business us, but am in the midst of launching a business that will be extremely dependent on availability of email (esp on Fridays!). Those of you using Gmail: What are Pros and Cons of using gmail for your business mail, spoofed from johndoe@yourbiz.com, as opposed to dedicated mail on your DH business account?

how can i know if i’m affected? i cannot find the link

Hi Guys,

No email for 2+ hours now. I’ve got a newsletter to publish and folks can’t send me editorial material via email for the newsletter. This is really a problem. Don’t forget to provide credits when billing time comes around!

this is insanity!! almost 2 hours!! stop the play and solve!!

Ahhhh. Solved? 5 mins after my post? Niiice.

Really need this to be resolved – I am waiting on some information for deadlines that happen today – any way we can access any of the info?

Yosved Villar lopez – log into your web panel through https://panel.dreamhost.com. In the upper right just below the logout button is a link that says Account Status. Click it. In the window that comes up at the bottom it shows your email server. If it is homiemail-sub5 then you are affected.

This creates a serious problem we would appreciate the service restored ASAP, thank you
—-Esto nos crea un problema grave les agradeceríamos restablezcan el servicio ASAP, mil gracias

it works now… (for me)

I’m really disappointed to be going through this….

PEOPLE. It’s shared hosting — if your business is SO critically important to the survival of humanity and your life will surely end if you cannot send/receive mail…

PONY UP AND GET DEDICATED HOSTING/SERVER

(And if you’re waiting until the last minute to send out a bid for a $2 million contract, that’s your own damn fault.)

Our email access has gone down at least twice a month over the past 6 months. sometimes briefly, sometimes extended. There needs to be better quality control. Email is how business gets done, and when it’s offline we’re falling behind.

Sean,

Just FYI I have a dedicated server with DH but email is handled on a separate system, so my email still went down. Having said that, DH service has vastly improved over past few months every way.

Do we know if the emails sent to our address during the outage will be retrieved? Has anyone received anything from that time in their inbox?

THANKS! working like a charm

You can send mail but not receive mail from our own domain, external as hotmail if received. Please review urgent

looks like the switch is fixed. Where is all the mail that came in whilst the switch was being fixed? Have you lost it?

Thank you! Good to be back in action again.

Boa tarde,

meu E-mail não esta funcionando, peço URGENTE, pois trabalho com e-mail.

http://www.coimo.com.br

Preciso enviar e-mails para clientes! estou sendo prejudicado.

att

Able to send email from domain, but still not receiving email from domains.

how long till mail comes through?

We can send but cant receive. What is a reasonable expectation before I should start to panic?
Thanks,

I lost emails during downtime!

Perhaps the system is still catching up, but I am still unable to RECEIVE any emails. I successfully sent an email via webmail, but I have yet to receive any of my test mails sent from other non-DH accounts.

Throw me in with the rest still not receiving any e-mails on my accounts on this server.

Able to send email from domain, but still not receiving email from domains.

Hey DH folks, just a reminder that some of us appreciate your work.

Perhaps these people don’t realize that the value proposition of a Crazy Domain Insane shared-type account means that maybe, just maybe, they shouldn’t be platforming high-value mission critical corporate shit on top of it! Get a dedicated host! And a room!

Just sucking up now… can you please make sure my missing emails from the past two hours show up!

Cheers,
Db

I would say that emails went missing. I am starting to get a few emails now but there are none for the three hour period that the server was down. Some are definitely missing.

Emails are now being sent but we are still not receiving all of out emails from other senders.

No incoming, seems I can send.

Will any mail not received during the downtime still arrive?

Thanks.

The above notification of this outage says “outage should not result in any lost mail.” Although I am now getting email, I clearly have a several-hour gap. Wondering if this downtime email is going to be coming through at some point, or is it, in fact, lost? Would be good to know.

My 5 email accounts are not working still…online chat is not working. Please advise asap. This is a big problem.

Cannot log into the panel or webmail. Can send but can’t receive.

Não estou conseguindo ENVIAR E-mail !!!!
Já está assim a 3 dias!

e-mail: godinho@coimo.com.br

Preciso de uma resposta!

att,

not resolved!

we can send but not receiving….!

I also seem to have this issue of not being able to receive emails. I was getting errors from my email client earlier today; now there are no errors, but also not getting any incoming messages. I’m able to send outgoing messages using my iPhone email client.

There was no mail lost during this outage. The mail queues are large and processing mail right now which is why you can send but not receive. Rest assured that your email will be delivered, but there is a delay while the MX machines mail queues are processed.

Our mail does not seem to be completely restored yet. I’m also having issues receiving messages. Just tried sending myself a test message after you all had indicated the problem was fixed. The test email sent, but I have not yet received it.

@Blair and everyone else experiencing issues with receiving mail please read my last comment:


There was no mail lost during this outage. The mail queues are large and processing mail right now which is why you can send but not receive. Rest assured that your email will be delivered, but there is a delay while the MX machines mail queues are processed.

We still can’t seem to send and receive — a bit troubling that status says “Resolved”
Can we re=open this ticket, please.

Thank you.

I can not send any mail or recieve as of 1:33pm EST.

this applies to an email being sent from a DH account to another DH account on different domains.

Oscar DH we thanks for clarify… we just hope and believe “there was no mail lost during this outage”… we are noticing this delay…

Mail is still down, cannot get any incoming mail.

Estou enviando e recebendo somente pelo webmail.coimo.com.br

Eu uso o Microsoft Outlook e não consigo enviar da erro de SMTP, por que ?

o webmail e muito ruim e não consigo organizar meus clientes!

PRECISO enviar e receber pelo Microsoft Outlook.

att,

So Dreamhost considers a 2 hour delay in receiving email a RESOLVED problem. I just got off a support chat, and was told that Dreamhost will refund the balance if you close your account. Over the last few years there have been multiple email outages every year. It’s time for me to go. Goodbye Dreamhost…

NOT RESOLVED, boys
i’m sending emails from another account until my accounts in dreamhost and nothing arrives!

I had no problems with homiemail last week while you were ‘fixing’ this, but this morning (1/28) all mail to me is bouncing from that server. NOT RESOLVED.

Hi, we are still facing with the problems with mail delivery, especially from online forms located on websites, When will these problems be solved?

Hello!
There is a problem not only on homiemail-sub5!
I have homiemail-sub4 and for more than a month there are problems!
When using the contact, no email can be received!
Please fix this problem!!!

Is this back again??? Getting POP3 failures all of a sudden….

It’s down – this is the third week in a row – getting IMAP drops too on Squirrelmail…

IMAP down again. Good times.

I am not receiving any emails now. I can send fine… but one-sided conversations with clients is not a good thing!

Multiple IMAP accounts on homiemail-sub5 are (barely) working – disconnections or extremely slow connections.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.5.1 – Maintenance and Security Release Available (RESOLVED)
2:53 pm

WordPress 3.5.1 – Maintenance and Security Release Available (RESOLVED)

Posted (January 24th, 2013 at 2:53 pm PST) by mir

WordPress 3.5.1 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Automatic upgrades, for customers that have them enabled, will begin shortly. If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.

This entry was posted on Thursday, January 24th, 2013 at 2:53 pm and is filed under Announcement, DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

21 Responses to “WordPress 3.5.1 – Maintenance and Security Release Available (RESOLVED)”

my sites got error 500 after this automatic upgrade!!

my site is down after this automatic upgrade (support ticket already submitted)

WordPress upgrade issues should be resolved now. We noticed the issue and had to rerun WP upgrades on some sites. If you are still having problems please contact support.

Is there any way to opt out of WordPress upgrades? I’d rather just do them myself =0)

February 12th, 2013 at 11:40 amFreemonSandlewould Says:

I can not even bring up dreamhost.com ! Also your database servers seemed to go down first!

Same here, Freemon. Completely unreachable from Lincoln, Neb. for mail, web, panel.dreamhost.com, etc.

Site and Dreamhost panel are down for me. Can’t even submit a ticket.

again all DH server is down, all domains are down, emails are down, jabber is down, web panel is down. whats going in

All servers are DOWN
All domains are DOWN
All account are DOWN
Panel.Dreamhost.com IS DOWN

all are down!!!!!!!!! … please again not!!!!!!!!!!!!!!!!!!!!!

Everything is down.
No email- No site- No Dreamhost Login- No Panel.
Not good. Some sort of update would be nice.

https://twitter.com/DreamHostCare says they are having network issues and are working on it.

again all DH server is down, all domains are down, emails are down, jabber is down, web panel is down. whats going on??????

seems again we have SCREAM HOST !!!!!!!

WE HAVE AGAIN NIGHTMARE HOST !!!!!!!!!

Either Los Angeles just got nuked by the North Koreans (they did promise something big…) after their test late last night, or a bunch of squirrels just got vaporized.

Oh, wait. Nope. Sorry, I was wrong.

Someone just tripped over a cord and unplugged the master power strip.

All servers are DOWN
All domains are DOWN
All account are DOWN
Panel.Dreamhost.com IS DOWN
dreamhoststatus.com IS OK (this is really sad)

so again we have SCREAM HOST AND NIGHTMARE HOST. AS SEEMS LIKE DH DONT EXIST. NO EMAILS, NO WEB PANEL, NO WEBSITE , NO JABBER

IS CALIFORNIA SUNK TO BOTTOM OF PACIFIC OCEAN?

SEEMS LIKE TECH GUYS OF DH TRIPPED OVER IN MASTER WEB SERVER ROOM AND MASTER CORD GET UNPLUGGED.

ANOTHER DISASTER

WE WERE MISSING ACTION.

DONT WORRY HERE IS SCREAM HOST BACK AND NIGHTMARE HOST BACK.

DREAM HOST AKA SCREAM HOST / NIGHTMARE HOST

Ya prendiendo el multitoma otra vez :/ ….. vaya !

SEEMS LIKE SOME STUFF START COMING UP, MAY BE CA START COMING OUT OF PACIFIC OCEAN LIKE A LOST CITY OF ATLANTAS

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – US-West Data Center (Irvine, CA) – Final Step – Wednesday January 23rd 6pm (4 hours) (RESOLVED)
4:53 pm

Network Improvement Series Maintenance – US-West Data Center (Irvine, CA) – Final Step – Wednesday January 23rd 6pm (4 hours) (RESOLVED)

Posted (January 22nd, 2013 at 4:53 pm PST) by Ralph

This Wednesday we will be finalizing our Network Improvement Series Maintenance on our US-West Data Center based in Irvine, California. The maintenance is set to start at 6pm on January 23rd and is scheduled to last approximately 4 hours. Once successfully complete, this will conclude our Network Improvement Series, which started back in May 2012, for the Irvine data center. During this maintenance, we will be moving public traffic from our old network switches to the newly installed switches. This will put all network traffic on our new network hardware. During the maintenance, we will be moving the public network in small batches. The plan we created will minimize the service impact from this maintenance as much as possible. As we progress through the maintenance we will monitor comments here and provide you with immediate assistance. We are excited to be concluding the Network Improvement Series in Irvine. Once completed we will be moving on to finalize our other US-West data center based in Los Angeles.

Update January 23rd 8:00pm PST: US-West Irvine, Ca data center maintenance complete! We now have this data center’s network completely working with our new upgraded hardware. This project started back in May, and for the Irvine data center, it is complete! This was an exciting step for us! On to our US-West Los Angeles, Ca data center next! That is however is for another day! We will keep you updated on our plans to fully complete the Network Improvement Series.

Update January 23rd 7:20pm PST: Old hardware is disconnected. We have some configuration changes to make on the new hardware. Once that is complete it’s a matter of monitoring to make sure that everything continues to work as expected.

Update January 23rd 7:10pm PST: We will next be disconnecting the old hardware and removing it.

Update January 23rd 7:03pm PST: And done! Just doing some final checking to make sure all is good. We saw no-impact from either support or from the comments!

Update January 23rd 7:00pm PST: DNS servers are going now. This should also only last a couple of seconds longer than the other moves, so it’s impact should still be hard to notice.

Update January 23rd 6:50pm PST: We will be moving our DNS servers for Irvine next. This has the potential to affect all customers in the US-West Irvine data center. We will make this change as fast as we can. We are about 5 to 10 minutes for attempting this part of the maintenance.

Update January 23rd 6:45pm PST: Moving along at a good pace here. No issues noticed or reported thus far.

Update January 23rd 6:35pm PST: We are moving networks quickly now and we just moved one of the larger ones with only a few seconds of interruption. So far the maintenance continues at a smooth fast pace.

Update January 23rd 6:30pm PST: Another successful network moved. Again, interruption was not noticed. So far this maintenance is going smoothly. Reminder that you can check to see if you have services in the US-West Irvine, CA data center in the panel.

Update January 23rd 6:23pm PST: First network moved successfully. Service interruption was less than a few seconds. We will be moving on to the next network.

Update January 23rd 6:05pm PST: We are starting with the first network. This is for some of our dedicate servers in the US-West Irvine data center. Service interruption should be quick. If you believe you are affected by this please leave a comment and we can check on it for you.

Update January 23rd 6:00pm PST: Our maintenance team is here and ready to get started. Comments are now open and we will provide timely updates as we progress through this maintenance.

This entry was posted on Tuesday, January 22nd, 2013 at 4:53 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Network Improvement Series Maintenance – US-West Data Center (Irvine, CA) – Final Step – Wednesday January 23rd 6pm (4 hours) (RESOLVED)”

How can I tell if my dedicated server is in the US-West Irvine Data Center?

https://panel.dreamhost.com/index.cgi?tree=status.dc&

the above link will direct you to your list of data centers.

Thank you!

Although my services are all hosted at “US – West (Los Angeles, CA)” could this be causing a problem with webmail for one of my accounts? I’m not able to log in even after setting user/pw via the DH panel.

Woody, this would not be affecting your account. If you create a support ticket and provide me with the ticket ID I can get someone to assist you asap.

My services are located at US – West (Los Angeles, CA) too and all my PHP mail forms and Cron Jobs are broken. I´ve created a Support Ticket (#5559821) but get no response until now… My site send activation codes by mail and i´m thinking if this is just a delay or a bug…

Teco, I have notified my team to assist you with your request. That is definitely not related to this maintenance. It might be related to a possible spam run. The support team will be able to assist you with information on your request.

I am unable to send emails, although I seem to be able to receive them. Is this being caused by the current system upgrade work?? It’s been like this for most of the day. Very frustrating.

Do you have a support ticket Bob? If it’s been most of the day then it is definitely not this maintenance. I am having one of my guys look into your issue. If you have not submitted a support ticket, please do so now, but I was able to get your domain from your comment submission and I’m having them check with that first.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Apache issues in US-West Data Center (Los Angeles, CA) (RESOLVED)
1:32 pm

Apache issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (January 20th, 2013 at 1:32 pm PST) by danwy

We are currently experiencing some issues in our LAX datacenter that are resulting in down websites for a small portion of our customers. This is in one of our datacenters that we are in the process of clearing out, so the number of affected customers is relatively small though. Email, MySQL and central services such as the web panel are all unaffected. We’ve found the cause of the issue and are already working on a resolution to restore the websites affected by this issue. We will update this post with more information shortly.

Update Jan. 20th 2:00pm PST: Upon further investigation we’ve found that most of the services affected by this are nearly empty, so there shouldn’t be much (if any) customer facing downtime. We are in the process of re-configuring all of the affected services to correct the problem and restore service to any domains still on these services. We will update this post again shortly with the status of how these updates are going. Please do contact support if you feel you are having problems related to this and we’ll be happy to investigate it further for you.

Update Jan. 20th 2:20pm PST: The updates that we ran on the affected servers have cleared up any remaining issues and all affected Apache services are now up and running. If you continue to have problems with any sites hosted in this datacenter, please let us know and we’ll be happy to look into it for you.

This entry was posted on Sunday, January 20th, 2013 at 1:32 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

33 Responses to “Apache issues in US-West Data Center (Los Angeles, CA) (RESOLVED)”

Since this afternoon our static IPs are giving us the standard Apache “It Works” and dynamic IPs are giving us a “bad_httpd_conf.” Additionally our FTP access is not working. We’re on the towns server. Is this related to the LAX downtime?

Thanks,

Adam

My website is down. Saying database error. Please help!!!

Now it is fixed.

Our site is not loading (or loading VERY slowly).

My site is either down or VERY slow.

still throwing database connection error

My site is also down, does not appear to be connecting to database.

My site is down. Not good. Important day for me!

Site completely down – not only is our site down, but every site on the server IP is down also.

Changed index.html to make sure it wasn’t our fault – it’s not.

Hope we don’t have to waste an entire day on this since we are in a dev phase on the site.

Please address this asap.

Thanks,
Moxy Ox Team

OK.. at LEAST once a week ALL my websites have been down.
Possibly DREAMHOST has the worst uptime in the industry.. AND I am paying a fortune for a VPS..
Switching hosts today after all these years.. enough abuse!

My website is also down and we have sent out an offer that leads to the website so critical that the website is up.

site down
error id: “bad_httpd_conf”

Most of my websites are down and not accessible via web browser nor ftp on any of our internet connections in the usa and canada. My websites keep going offline because of dreamhost problems and that’s not a beneficial business practice at all. We always pay for our services with dreamhost and dreamhost is consistently unable to provide the stable services we pay for and expect :(

as of right now, all of my sites are down. :(

My site is down as well – no FTP access through AjaXplorer. Our second website through Dreamhost is working but verrrrry slowly.

all my sites are down again

All my websites & mail have been down since Friday, no response from Support : (

My site is down. Still

All my websites are still down. No response from CS. – just hearing from my customers.

Our site has been down for the last 5-6 days this is unacceptable dreamhost.

It’s more then what there telling us this shit is

Not fixed. My site is still not working. Can ping and ftp, no www.

My site is down as well. Either loads very very slowly or not at all (server error).

Our site is loading very slowly and we get server errors frequently.

PHP not loading MySQL extension.

None of my drupal sites are connecting to the database. Is the issue here related??

A database error occurred while processing this request and can’t login to dashboard

main dreamhost.com site down…

My site has been down or significantly slow for several days at unpredictable times. According to the information I have, it’s hosted in the US-West Data Center.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail downtime with homiemail-sub5 and webmail (RESOLVED)
10:22 am

Mail downtime with homiemail-sub5 and webmail (RESOLVED)

Posted (January 17th, 2013 at 10:22 am PST) by Oscar

We are currently experiencing problems on one of the file servers that the mail machines mount to. Because this is where the mail files are stored it is not possible to access customer mail while we get this corrected (mail will still be accepted for delivery and written to disk once the file server is back up). Our system administration team is diligently working towards a solution. Again, this only affects users who are hosted on homie-sub5. You can confirm whether you are affected by clicking the ” Account Status ” link here http://panel.dreamhost.com. Scroll to the bottom and you will see a field labeled ” Your Email Server “. If its set to homiemail-sub5, that would explain any mail problems that you are encountering. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

Update Jan. 17th 10:45am PST: This issue has now been resolved and you should be able to connect to your email via webmail or email client. There was no mail lost during this outage and we are currently reprocessing all messages received during this time. Thanks your patience while we resolved this issue. – Oscar

This entry was posted on Thursday, January 17th, 2013 at 10:22 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

14 Responses to “Mail downtime with homiemail-sub5 and webmail (RESOLVED)”

Yes, all email accounts no works.
please, make a fix a soon as possible

I’m having issues with homiemail-sub4

Perhaps it’s affecting more than homiemail-sub5

Any idea of ETA for a fix?

My email (on homie-5) is working sporadically now.

this always seems to happen to this server. totally insane and frustrating.

sporadic works on homie-5

Any reason why all the websites we host would have gone down also? Error stating “Database Error: Unable to connect to the database:Could not connect to MySQL”

Too bad cod:nocito isn’t working either…

Your company is a fucking joke we have had our site down for 4-5 days when the hell is it going to come back online what a joke of a company your size get off your lazy asses and fix the god dam thing….

whens the “Database” errors going to be fixed and addressed?

Sometime this month would be nice………

On the contrary, I find the mail to be more stable than most and the web site hosting has been rock solid. Kudos aside, I have been notified that my email (on homie 5) seems to have been hacked. Could there be a connection?

I can’t log in to my email, any clue whats happening with homiemail-sub5 and webmail?

I’m having the same issue. Didn’t have any of the problems listed above on the 17th, but my email server is homiemail-sub5 and only today, seems to have stopped working across all pop3 and webmail servers for all domains. The email I am posting here with, does not receive email, so won’t do any good to reply to it until the issue is fixed.

Also have no email access this morning as DigiP reports. Does not work by web interface, smartphone, Outlook. I’ve submitted a trouble report. Just stating it here for general information. Can only see updates from DH via the individual trouble history page.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web Panel and Webmail Down (RESOLVED)
11:12 pm

Web Panel and Webmail Down (RESOLVED)

Posted (January 15th, 2013 at 11:12 pm PST) by Steven

I am sorry to report that we have run into trouble on our central database server. This is causing downtime for the Customer Web Panel, Webmail, and our internal systems. We are actively investigating this issue and hope to have it resolved very soon. Additional information will be posted as it becomes available, or when this has been resolved. Thanks for your patience as we work to get everything back online.

Update Jan. 15th 11:56pm PST: As of 11:40pm all services have been restored. Our admin team found the Panel and Webmail machines were experiencing connectivity issues. The were able to find the source of the problem and resolve it. No customer services (web, mysql, or email) should have been affected. This was isolated to just the Web Panel and Webmail services. Once again, we thank you for your patience and apologize for any trouble this may have caused. -Steve

This entry was posted on Tuesday, January 15th, 2013 at 11:12 pm and is filed under Control Panel Outages, General Outages, Webmail Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Web Panel and Webmail Down (RESOLVED)”

Goodmorning from Europe – no its not can’t get into my mail – again!

I’m trying to get a db configured, I have developers waiting, and the panel is continuously going down… very frustrated.

Looks like still the web panel and webmail is down, please fix it ASAP

Webmail working now.. From Sri Lanka

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DreamHost control panel down (RESOLVED)
4:46 pm

DreamHost control panel down (RESOLVED)

Posted (January 15th, 2013 at 4:46 pm PST) by Seohee

The DreamHost panel (https://panel.dreamhost.com) is currently unavailable. Our administrators are working on finding the cause, so we’ll make sure to provide you with further updates shortly.

We are sorry for the trouble, and appreciate your patience at this time.

Update Jan. 15th 5:07pm PST: The issue appears to have been due to a database lockup, which then resulted in a pile-up of connections from the panel machines. This pile-up then caused the panel machines to run out of memory. This has been corrected, and our administrators are working on putting a solution in place to prevent this from happening again in the future.

The panel is up and running well at this time. Again, we appreciate your patience while we got this worked out, and are sorry for the trouble this has caused. – Seohee

 

This entry was posted on Tuesday, January 15th, 2013 at 4:46 pm and is filed under Control Panel Outages, General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “DreamHost control panel down (RESOLVED)”

Hopefully not too long, need to setup a new site and some email.

I’m more worried about all my websites running really slow in appling :\

Seems to be back up

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Outage In Our Irvine Data Center (RESOLVED)
12:22 am

Network Outage In Our Irvine Data Center (RESOLVED)

Posted (January 15th, 2013 at 12:22 am PST) by Steven

We are experiencing a bit of a network flap in our Irvine data center. Our system administrators and network engineers are currently working to identify the cause and resolve it. We will update this post as soon as further details are available! Thank you for your patience.

If you are not sure which of our data centers you are in, you can look that up in your DreamHost panel under Status / Data Centers. Please note, this only affects customers located in Irvine.

Update 1/15 4:10am PST: All networks are functioning normally once again, and all sites should be back up and running as usual. We sincerely apologize for the length of time your sites were down; we were diligently working to restore connectivity as quickly as possible. We will post a follow-up later today with more details. Thanks once more for your patience. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Update 1/15 3:50am PST: All our networks appear to be fully operational again. We are currently working on restoring the redundancy between our data centers and looking at any remaining servers issues that were caused by the outage. We should have a complete resolution soon, and will continue updating this post with any further information. Again, we apologize for the issues this caused and thank you for your bearing with us!

Update 1/15 2:50am PST: We are getting closer to having this resolved and things are looking up. Throughout our troubleshooting process, we discovered that the outage was affecting our ability to automatically handle DDoS attacks. Fortunately, we are almost back to normal traffic levels, but are still experiencing a few remaining issues. As a reminder, you may notice intermittent downtime as we continue to troubleshoot and address the cause of the outage. Thank you, once again, for your patience while we work to resolve this.

Update 1/15 2:05am PST: We have found a few issues that had to be resolved by rebooting some of our internal network hardware. As a result, many impacted networks are back online. However, if your site is currently in Irvine and working, you may notice further intermittent downtime as we continue to troubleshoot and address the cause of the outage. Thank you, once again, for your patience while we work to resolve this.

Update 1/15 1:00am PST: Our network and admin teams are hard at work troubleshooting the cause of the outage. We sincerely apologize for any inconvenience this is causing. Further updates to come as we have them. Thank you for your patience!

Update 1/15 12:30am PST: We are still in the process of looking into this and will continue posting updates regularly with information as we have it.

This entry was posted on Tuesday, January 15th, 2013 at 12:22 am and is filed under General Outages, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

157 Responses to “Network Outage In Our Irvine Data Center (RESOLVED)”

Currently affecting me I think

Affecting me – site is down. Glad to know it wasn’t something I did.

My website is currently down. The IP address that gets resolved when I ping http://www.abraxasenergy.com is different from the IP shown in the dreamhost panel. I have a VPS with a dedicated IP address. Hope that helps.

January 15th, 2013 at 12:35 amJ. Bart Henthorn Says:

It’s taken all my domains offline. Understandable when things like this happen, but still frustrating that it takes them all down at the same time. (I should probably review the old saw about “eggs” and “one basket” .. huh?)

My site is down. This happened for the third time in the past 30 days.

All my sites are down. On my dedicated server. The bloody Irvine datacenter again.

My all site are down…

come on lads get it sorted there’s chattin to be done

Just wondering if there is a way to confirm if my sites are on this server in the Dreamhost dashboard? Both sites are down for me now, so I’m guessing so.

My site is down too

Thats terible very unprofesional!

I have multiple websites hosted here and they’re all down.

The same things for me.

Our dedicated server is completely down. Everything is dead

All sites down!!!! omg!!

All of my websites are down too.

All my sites are down now

Me also, any good news?

Is this a memorial event for Aaron Swartz?

All my sites are down too

All my websites are down.

Thank god I am not the one responsible for this.

Please DH, fix this outage soon.

This is ridiculous. Our site is a tennis website, and the Australian Open is currently happening right now. Our site is completely down during peak hours.

All our sites are down – sounds like a network-wide failure in Irvine.

my site (gaygdl.com) is hosted by Dreamhost and now it is online, can you help?

All my sites are down too…

When will this be fixed (Aprox)?

Mine is also down. After 14 hours of work, clicked for final update and then…. nothing. Terribly unsatisfying.

WTF Dreamhost… WTF. Your reliability levels are going downfall, near Yahoo Small Business

January 15th, 2013 at 12:48 amCyril Washbrook Says:

Kat writes: “Just wondering if there is a way to confirm if my sites are on this server in the Dreamhost dashboard? Both sites are down for me now, so I’m guessing so.”

In the panel, scroll down the left-hand navigation until you see “Data Centers” – it’s a link under “Status”. You’ll see a list of the facilities where your services/databases are being held. Chances are that if your sites have gone down, it’s because of the Irvine outage.

My site is still down. I would appreciate if you could give an ETA or admit if you don’t know what the problem is. I’m a webmaster and would need to take proper precautions depending on when the site will be back up.

ETA for a fix? I have a big presentation with my newly redesigned website at 11AM Central!

Is my data safe? The third times in the last 30 days? I’d better change to another server.

I thought there was supposed to be a redundant/backup network link in place for Irvine after all those outages last year? Wish I could move my dedicated box to another DH data center.

And we’re back. Good job, guys.

Site down, no response to ping requests !!!

All my site down!!! My pupils need it to work! Please repair it :-(

January 15th, 2013 at 12:52 amJ. Bart Henthorn Says:

40% restoral so far. 2 down, 3 to go.

Some of mine are down still. Trust to be the most important ones.

Online again! Thanks!

Now it works for me

At least they realized that there was something wrong almost the same minute the shit started to hit the fan. And they made the post like 1-2 minutes after it actually started to happen. I know because I have uptimerobot monitoring my sites. So thats quite impressive at least. Hmm, lets also hope they can sort this out with the same quick response time.

Mine are down two… All of them.

Isn’t it expected to have a work around process to fallback on another datacenter?

I was going to set a client’s website online right now. What do I look like now?

We are on top of the line dedicated servers here. 11 of our sites have been down for about 2 hours now. Wtf?.

Currently affecting about 10 of my clients sites… hope this isnt going to take long as 80% of them are ecommerce sites.

affecting MySQL.

Time is money…

I am afected. My company is stopped! Panic!!!

@Balint– I don’t know. I was down for about 12-14 minutes before there was any confirmation from DreamHost.

Even all my websites are down.. I am on VPS..

This is really awesome. My VPS is on a server at one of the data centers and my MySQL server is at the other data center. So no matter where you go down, you take me with you. Brilliant!

One of our sites is back-up … sounds like the’re getting on top of it

this site and others are all down

We’re down too! (Also glad that it’s not our ISP futzing things up like they seem to)

My site is back up, for what it’s worth. Has been for a few minutes.

I had a hard time finding out if dh was down because there are some dh status sites but apparently they are old- I finally the answer on twitter.

Shouldn’t you delete these?

http://status.dreamhost.com/
https://twitter.com/dreamhoststatus

It would probably save you some headaches

My site is down too. Right in the middle of doing a website news update. Hope it doesn’t take long.

January 15th, 2013 at 12:58 amJ. Bart Henthorn Says:

Yeah, I’m back to 20% restored. One of those that had returned is now gone again. Sounds to me like someone cut the red wire instead of the blue one. 8^)

All the sites down … WTF ? … Has dreamhost no redundancy ?

Interesting, DH spotted the outage before our (pingdom) monitoring did – which is good, and our VPS site is now back on-line. (Share server sites still down).

My Data Center is located in US – West (Los Angeles, CA) as mentioned in DreamHost Panel and affected to all my websites (e.g: http://thefoodiecrowd.com)

Seems to be working now.

my web service also effected

Well that’s crap but upside is cloudflare has kicked in on a few sites and they are still working including http://www.losangeles.net.au/ which is great for that one site I setup the rest of them are down…

The only thing I am paranoid about – apart from the fact that I have the most expensive dedicated machine here and regardless all my sites are down – is that tomorrow they have a 4hours long network maintenance scheduled….I am scared.

I haven’t felt worst since major 2003 NYC blackout! You’re right Nicholas, time is $$$

My site (http://diskwave.barthe.ph) located in “US – West (Los Angeles, CA)” datacenter, on machine “dodgers” is also down. It seems that HTTP server is down, even though I have no problem accessing the server via SSH.

Still getting error

—————–

Error:

Can’t connect to database!

MySQL said:
Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110

How do I know what center I am served from? Instructions to look at the right under the account status. I don’t see where it says Irvine anywhere. So, how do you tell?

So far 30 minutes downtime. Lets go guys!

my website is being affected…

My site is down also. As someone else has stated, this has happened several times in the last several weeks. I was on the LA data center and it happened there. Then I switched to the Irvine data center and it’s happened here. Can’t afford to have this happen. Up until now, over the last several years DreamHost has been great.

Error establishing a database connection

I’m back up total of 54 minutes of downtime.

Happened just after I loaded a couple of big data tables, thought I’d broken my server, scared the s*** out of me :)

My VPS is still down. I recently signed up to Dreamhost because of the 100% up-time guarantee thinking that adequate redundancy would be built in. I hope that this sort of downtime is very rare or I will move on. I always find it amusing when webspace goes down and yet the hosting companies main website always appears unaffected. I always think: why don’t you host me where you host your own website!

To FOODIE CROWD – everyone here wishes to showcase their websites, but it involves REALLY A LOT of imagination, since most of us if not all don’t know each other and have never seen each others sites. We are on the same Titanic at this point… Lets hope we are not going to freeze to death… jk…

Database connection error.
Admin, we humbly beg your kindness… Help.

This is happening so often.
We’re going to move our site out of your servers. What a pain!

My site is down and students at the university were in the middle of doing their homework and now can’t access any of the material from the course website.
This is very frustrating

I am very concerned that my site is offline

January 15th, 2013 at 1:11 amJ. Bart Henthorn Says:

I have to compare against my previous web hosting service (Dotster). Had this sort of outage hit them (which it did ALL the time), they would have not posted anything, not said anything .. and if they chose to respond to a support ticket about the issue, the response would have come about 5-10 business days later saying “we’ve checked and everything seems to be up. What are you concerned with?” (SLAM – ticket closed!) In other words, yeah, it sucks .. but compared to where I was? Dreamhost wins by a wide margin. PS: Back to 40% restored – still 3 sites down.

money, much money!!!

WEB page work ok. but MYSQL don’t work. i recive this mesage: MYSQL too many connections!

My website is down. It is peak time for us in India.

Where’s the dream ? Maybe in your dreams, dreamhost company !!!

This is terrible; still down. I think i’m going to have to consider switching hosts, as this is not the first time.

Affecting me – site is down!!!

Don’t tell me that this is a promotional event for cloud flare service

My website is down :(

My site is up now :)

Oddly enough, my system has started responding to telnet on port 80, and I can send a GET request, and I quickly get a bit of the response, but then it hangs for 60 seconds before sending the second half of the response. Before 10 minutes ago, I was getting no SYN+ACK in response to my SYN. But something is still having issues.

Our site is also down

We’re i the middle of a marketing campaign, we’ve sent thousend of mails to our customers, and guess what…no website available, we look like clowns

YES!!! I’D LIKE TO HOST MY SITES ON THE SAME SERVER AS DREAMHOST.COM!!!
THEY ARE ALWAYS UP, NEVER AFFECTS THEIR SALES, WHY IT SHOULD OURS???

This is painful. I was testing the VPS servers and I’m done with this.

Can’t Connect MySQL Server

….may be nightmarehost? :(

Down here as well in the Czech Republic. Same as Nicolas, just launched a marketing campaign. Not a happy bunny. Thinking of changing hosts.

NICK! EVERYONE LOOKS LIKE CLOUN HERE NOW…

My sites are back up :-)

CLOWN THAT IS

All sites down!!!! OMG!!!!!

I hope when the server does come back online, none of our information will be lost??? This is nervewracking!

Please hurry… this down time is costing me money and clients

Got this message :

We were unable to verify any server-wide problem related to
the outage you’re experiencing. If you submitted a support
ticket, we will be with you soon to help resolve this issue.

We’re very sorry for any inconveniences,
The UnHappy DreamHost Notification Team

ALL SITES ARE STILL DOWN … and they are unable to verify !

While the best developers are here now…
.
**************************************************
###################################
.
ANYONE KNOWS TALENTED DESIGNER?
.
???????????????????????????????????

All my websites online again. Thank you guys! :)

Like the name says… I hope all this is happening in the dream.

all site down… oh oh..http://www.vietnamtourbooking.com and halong-bay-tours.com

My all site are down…

我们所有建在你们服务器上的网站,全部无法打开了,损失很大!希望可以尽快恢复!

hope all sites will backup soon
sites down from 1 hour xd <

Getting a Database connection error. I have already moved one site to another host. Just hope this gets sorted out quickly.

All my sites are down (romasocialmedia.com & tellyourcountry.com). Damn it! Please, help ….

GUYS I THINK IT’S GREAT TIM TO GROW A BEARD!!

Oh dear.I have 4 storage account can not be accessed.Please handle this problem quickly.All my websites stopped working.

You guys really need to get your $%&&**(* together, it’s not like we’re not paying for this freaking hosting arm and a leg $30+ monthly is not cheap.
Get this to your bosses and don’t let outages ruin our lives.

Want to be listed as the top dog on WordPress? well fix it for crying out loud and start offering free *%&*^((# every time this happens if you want to keep customers!!!

All my sites are gone down. Please fix it asap.

It is totally out… fix it for me

now all my sites are back! great and thanks!

Now they works! :-) Thanks

January 15th, 2013 at 1:40 amJ. Bart Henthorn Says:

Appears to be 100% restored for my sites. Granted, it’s gonna be fun trying to recover the lost transactions that evaporated while things were down, but it still miles better than Dotster! (Can you tell I have an abiding dislike for Dotster? LOL)

Downtime:
2013-01-15 07:34:06 UTC – 2013-01-15 09:26:29 UTC = 112 minutes.

Not great but in the last 6 months DH haven’t been too bad (for me) so i’ll let them off this one slip!

Good job guys ! It’s come back !

I am new to DreamHost and have a VPS account. Can those who have been here a while tell me how often this happens? Need to assess whether to stay or move on. Thanks!

Dreamhost stuffs up yet again. Once again, my site is offline costing me money. Had huge issues with Dreamhost over the last 12 months, trying multiple hosts which were poorly managed for resources on the host servers (continually crashing and timing out because they cant resource manage their infrastructure) – Sad to say the Dreamhost of today is not what it was 6 years ago. I’m jumping ship after 7 years with these guys. I’ve had enough!!!! Anyone know any “good” hosting providers I could look at?

All my sites ara down too. Thanks for fixed asap…

Could not have come at a worse time. Just launched new admin site for client and when the staff arrived at work, everything is down. FTP and Admin panel still up.

We are up!!!!!!!!!!!!!! Thank you

My websites are also UP now…

Anthony: I used to host with pair.com a while back and found them really good. Might be worth checking out. I am thinking of going back to them.

Shared server sites back up now as well as the VPS site up earlier.

Back online again!

Both of my websites have been down since yesterday afternoon. This has become far to frequent of a problem and is having an extremely negative effect on my online companies sales. I am currently paying DreamHost while I lose income because of your issues. 10 hours and counting that my online store has been shut down. This is totally unacceptable.

so yeah… like any other thing in life, sometimes things break. thanks for keeping us updated, and allowing all these ppl to vent, its quite hilarious. dreamhost rocks, and yes we understand sometimes… it happens.

Sure… but how often does it happen?

This happens too often and we NEVER get compensated. not even discounts on the high $50/month bill. I a FED UP with these clowns.

Our system online again!!! Thanks guys!!
^_^

What is going on! this is happening very frequently now.

Please, while you’re hard at work fixing this issue, don’t forget of the caeneus mysql server, which is stuck with “Too many connection” error for more than 5 hours. This is quite unacceptable.

Back up. Thanks for the fast recovery Dreamhost. Appreciate your quick action and hard work!!

Impossible to acces to my dbases placed at arachne!

The vps server: barriodelcarmen having problems for two days.
Very slow or “Database connection error (2): Could not connect to MySQL.”
In all sites. SITES DOWN

@dude relax – I can understand a couple of blips over a 12 month period. But I’ve been offline for days + for me and I’m talking continual tickets submitted and needing to move hosts all the time. I’m usually a relaxed guy, but it just happens so often with these guys now. Fair enough if you’re running a blog or something, but an ecommerce website which is bringing in money to support family/mortgages etc, its simply not acceptable.

I’m still down, 2 hours later. hope it gets resolved soon!

没事没事 that’s nothing

Please have someone look at the issue with the with the arachne:caeneus database as soon as possible. The outage has been marked as resolved, but I can assure it isn’t. The outage started almost 6 hours ago!

You guys just moved my domain to Steward, 2 days later, datacenter down! what about plan b? backup connections, etc, etc.

Sign up not a month ago and this is what i get …I want my money back

Sites are back but some plugins failing – for me so far, MapPress and TheThe Tabs and Accordions :(

thanks

thanks for letting us know what’s going on down there
greatly appreciated

January 15th, 2013 at 4:17 ambuythiscomputer Says:

You should offer cloud to everybody !!!

Our websites should never be down !

I’m still having problems. First the Dreamhost said that the problem was now the hacker problem is the DataCenter .. Please solve the problem, I depend on my site to work.

Mysql server is very slow.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web Panel and WebMail inaccessible (RESOLVED)
4:35 am

Web Panel and WebMail inaccessible (RESOLVED)

Posted (January 12th, 2013 at 4:35 am PST) by patrickm

We regret to post that we are having issues with our central database server, which is causing downtime for the Customer Web Panel, WebMail and our internal systems. We are actively investigating this issue and hope to have it resolved very soon. We will update this post with more details if we have them, or when the issue has been resolved. Thanks for your patience as we get everything back online.

Update 4:46am – We have resolved this problem and both the Customer Web Panel and WebMail access has been restored! Thank you for your patience as we made quick work of this outage! -PatrickM

This entry was posted on Saturday, January 12th, 2013 at 4:35 am and is filed under Control Panel Outages, Email Issue, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Web Panel and WebMail inaccessible (RESOLVED)”

Hello guys,

Any news about the outage of the panel? Have it been out for 12 hours now?

Thanks,

Lucas

Just as I clicked save to do a big switchover on my site. Frustrating!

Thanks for getting a status post out so quickly though. I wish you all luck.

Site says resolved, I still got errors, email from tech support says it will be resolved shortly.. Post by PatrickM says resolved… Who to believe…lol

I am still getting error ” you must login To access this page” after I typed in the username and password at webmail login page. Issue is not resolved. What’s going on?

I still cannot access panel.dreamhost.com — What is the ETA on getting this problem resolved?

Seems like this is an issue again. Right in the middle of the day.

January 15th, 2013 at 4:41 pmWalter Gardeazabal Says:

Not the middle of the day for me, but same here, panel inaccessible… any news/update?

Same here – can’t access panel, although all sites seem to be up.

More info please…

need an ETA here guys, this is being 24 hrs and counting….

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance in US-West Data Center (Irvine, CA) Wednesday January 16th 6pm (4 hrs) (RESOLVED)
4:19 pm

Network Maintenance in US-West Data Center (Irvine, CA) Wednesday January 16th 6pm (4 hrs) (RESOLVED)

Posted (January 11th, 2013 at 4:19 pm PST) by Ralph

As part of our continued effort to improve our network services, we are conducting hardware maintenance in our US-West Data Center (Irvine, CA) on Wednesday January 16th 6pm. We don’t expect any customer-facing service downtime, but changes made in the DreamHost panel (modifying or creating domains or users, for instance) may be delayed. We are scheduling a wide four-hour window to complete the maintenance and the pre and post testing and to cover any unforeseen complications, but the actual process will likely be much shorter.

You can check if you’re potentially affected by this on the DreamHost panel .
If you have any questions about this before or during the process, please let us know by contacting Support We will be happy to help. During the maintenance, we will also leave the comments open and will be monitoring them. We will also provide periodic updates during the maintenance window.

We apologize for any inconvenience this may cause and appreciate your continued patience while we further improve our infrastructure.

Update Jan. 16th 10:25pm PST: Our tests are complete, as is the maintenance! We appreciate your patience.

Update Jan. 16th 10:25pm PST: Still going well and the testing continues.

Update Jan. 16th 9:30pm PST: Everything’s looking good at this time, but we’re still in the testing phase.

Update Jan. 16th 8:30pm PST: The re-routing of internal traffic is complete, and we’re now in the testing phase.

Update Jan. 16th 7:29pm PST: Still going well! We’ve moved the firewall functionality to our new hardware and the tests look good. We’ve now moved on to re-routing the internal traffic. Again, this should not cause any customer-facing problems.

Update Jan. 16th 6:57pm PST: So far so good! We’ve finished testing the connection, and are moving onto the next step which is moving the firewall functionality to our new hardware and continue our tests.

Update Jan. 16th 6:36pm PST: At this time we will be experiencing some delays in service setup. This will only affect new services being created from our panel and will not affect your live hosting. We are starting with the first step of our maintenance which involves switching our data center communication traffic to reside on our new upgraded network hardware, and then testing the connection.

Update Jan. 16th 6:12pm PST: The maintenance has begun! We’ll provide you with any necessary updates until it’s complete.

This entry was posted on Friday, January 11th, 2013 at 4:19 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Network Maintenance in US-West Data Center (Irvine, CA) Wednesday January 16th 6pm (4 hrs) (RESOLVED)”

Sites on mensa in the LAX data center appear to be down.

Mensa is not in the Irvine datacenter so is not affected by this maintenance.

sadly my phpmyadmin import feature has stopped working..

import.php: Missing parameter: import_type (FAQ 2.8)
import.php: Missing parameter: format (FAQ 2.8)

worked just a few minutes ago… must be related to this.

@lyndon

Sorry to hear of your troubles. Your VPS is actually not in the Irvine datacenter. Please contact support if you are experiencing trouble with phpMyAdmin.

Where is the Nayda server issue? Why is it not posted in DreamHostStatus.com? My sites were unstable since this morning. I submitted a trouble ticket and did not receive a response for 6 hours. Now 4 of my sites are down/removed for almost an entire day. And all my sites have a coming soon banner. All this because you can’t properly monitor your NAYDA server in Irvine, Ca. What’s worse, you never posted any of this in DreamHostStatus.com and never tweeted any of this.???

@Drew: The status updates for nayda are posted in your panel (panel.dreamhost.com), so that only the people on that server see the updates. Single-machine issues are not posted to our status blog.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Email Hardware Maintenance in US-West Data Center (Los Angeles, CA) Tuesday January 15th 8pm PST (4 hrs) (RESOLVED)
2:57 pm

Email Hardware Maintenance in US-West Data Center (Los Angeles, CA) Tuesday January 15th 8pm PST (4 hrs) (RESOLVED)

Posted (January 11th, 2013 at 2:57 pm PST) by Ralph

As part of our ongoing network improvements, we will be conducting email hardware maintenance in our US-West Data Center, Los Angeles on Tuesday, January 15th at 8:00pm PST. This will affect less than 10% of the email service customers in that location. We are scheduling a four-hour window to complete the maintenance, perform post maintenance testing and cover any unforeseen complications. The actual downtime to your email service is expected to be much shorter. During the maintenance we will be updating this post to provide you with the actual status of mail service throughout the maintenance. When service is interrupted, you will not be able to log in, check email, or send email, also any mail sent to you will most likely be queued by the sending machine. As soon as the maintenance is complete, and the sending machine tries to re-send the message, queued mail will be delivered successfully.
You can check if you’re potentially affected by this on the DreamHost panel.
If you have any questions about this before or during the process, please let us know by contacting Support. We will be happy to help. During the maintenance, we will be allowing commenting and will be assisting you here as best we can.

This post will be your best source for periodic updates and information regarding this maintenance window.

We apologize for any inconvenience this may be causing and appreciate your continued patience while we further improve our infrastructure.

Update Jan. 15th 08:57pm PST: The file servers are all now operational again. Everything is working properly so we are closing this maintenance window. If you have any problem or then please let us know by contacting Support. Thanks!

Update Jan. 15th 08:34pm PST: The file servers have been moved to their new location. Our admin team are powering them back on at the moment so they can update the new network settings.

Update Jan. 15th 07:55pm PST: We’ve just opened up comments on this post and are about to get underway. We’ll be monitor the comments for issues. We appreciate your patience during this update.

This entry was posted on Friday, January 11th, 2013 at 2:57 pm and is filed under Data Center Maintenance, Email Issue, General Outages, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance for US-East Datacenter – Maintenance To Resolve Mysql Issues (RESOLVED)
7:36 am

Network Maintenance for US-East Datacenter – Maintenance To Resolve Mysql Issues (RESOLVED)

Posted (December 17th, 2012 at 7:36 am PST) by Justin@DH

Our apologies for the issues with mysql connectivity in the US-East Datacenter this morning. Our onsite datacenter staff are performing an urgently needed maintenance to replace a defective network device. Staff are on site and we are expecting the maintenance to be completed shortly. We will update this post when we have further details on its completion.

Update Dec. 17th 7:37am PST: The maintenance has completed, the hardware has been replaced. We are watching the systems to make sure all is fully functional at this time.

This entry was posted on Monday, December 17th, 2012 at 7:36 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

5 Responses to “Network Maintenance for US-East Datacenter – Maintenance To Resolve Mysql Issues (RESOLVED)”

This has been going on intermittently all day. Fix-Up!

@Jay: That is the plan, the network component that is causing the issues is being replaced now.

@Daveline: 10 to 15 minutes at this time.

Missing argument 2 for wpdb::prepare(), called in /home/wendersouza/hotelpousadailheus.com.br/wp-content/themes/welcome_inn/library/tfuse_framework/functions/core.php on line 29 and defined in /home/wendersouza/hotelpousadailheus.com.br/wp-includes/wp-db.php on line 990

December 17th, 2012 at 7:49 amThiago Colebrusco Says:

It seems that my MySQL problem was solved. Does the maintenance finished?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.5 Major Release Now Available! (RESOLVED)
12:47 pm

WordPress 3.5 Major Release Now Available! (RESOLVED)

Posted (December 11th, 2012 at 12:47 pm PST) by mir

The newest version of WordPress, 3.5, is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

This is both a feature and maintenance release, and is highly recommended for all customers! Its new features include a new and improved media workflow, new default theme, and a Tumblr importer

For more details, you can check out the release notes, here, with further details available in the codex.

If you run into issues after an upgrade, as a first step, take a look at this thread on 3.5 in the DreamHost forums, and the 3.5 master list at the WordPress.org forums, where issues relating to upgrades are being tracked.

The update will be automatically installed for customers who have automatic upgrades enabled. You can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If after upgrading, you’re having issues with the ‘Add Media’ button and have Google Pagespeed enabled, check out this post for further information!

This entry was posted on Tuesday, December 11th, 2012 at 12:47 pm and is filed under Announcement, DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

40 Responses to “WordPress 3.5 Major Release Now Available! (RESOLVED)”

Dreamhost’s web panel is down.

Be aware with updating to 3.5

Drag and drop widgets wont work.

Drag and Drop does work, but there is plugin problems which interfere.

If you use the Easy plugin be sure you are running version 0.7.3 of it.

Any plugin that adds to the page’s JS code will also cause a problem, so turn off plugins until it works and please report which plugin is causing the issue to WP if you can.

Thank you!

ftp server files look ok

web panel is down?

December 12th, 2012 at 12:00 pmArturo González Says:

MySQL is Down!!!!

My ruby rails app is down too.

Sites down too, what happened!

our website has been down all day today with no response from Dreamhost. I can’t get into the wordpress dashboard and it gives me a 404 error. I’ve been extremely frustrated by the total lack of response from dreamhost.

Our site is down tonight, I can’t say if it was down at all today however.

screw u guys. snowbird is filthy slow even with super cache. You all ought to be ashamed of yourself.

Seeing as no one from dreamhost is contacting ANY of their customers regarding the outages today nor are they responding to urgent tickets, I will let you all know on their behalf whats going on. Today Dreamhost has convinced me that it is time to look for a new server for our websites. Something as important as website outages, you would think that they would have someone let us know what is going on. Not the case. I have not heard a peep from these guys nor are they posting any updates or saying that anything is wrong.

Anyone know what the hell is going on? I feel like the air raid sirens are sounding but no one is telling us what the problem is.

@Cam in case you’re switching here is my shameless linode referral code http://www.linode.com/?r=0ca24dc2e9a2b21cba8053674e4b570cae8fed7b get a good vps and forget your troubles. Learn to install webmin and thats all you need. Keep dreamhost for emails as setting up email is difficult and google apps is no more.

as of now the sites are inaccessible. i hate DH. I wish so much I had not setup so many sites here. All I wanted is an incubator and that itself DH fails at very much.

4:03 pm friday, dec.14th.
MySQL servers for all of my sites are down. No response from Dreamhost and no messages on Status page. It sounds like other people have been having the same problem over the last couple of days

A response from DH would be reassuring…..

My databases just went down while working on a site. What the hell is going on DH??

Mine is down too =/

Someone has started a thread on the DH forum:
https://discussion.dreamhost.com/thread-136644-post-159321.html#pid159321

maybe we can all rant there and ask for some info

DH just FIX IT! my sites are all down.

my regular sites are not down but my wordpress sites are. i literally was jus ton one 2 hours ago and it was fine.

then i get an email from dreamhost saying that they tried to upgrade my wordpress but the database isnt there?

I WAS JUST ON IT!!!

Looks like my mysql database sites are down as well. I’ve opened up a ticket. Tried to do a live chat, but they must be swamped with calls.

Same here, mysql database connection fails.

Databases appear to be back up. Fingers crossed.

MY dedicated server is down againnnnnnnnnnnnnnnnn very bad service

Page Speed BETA (used by Dreamhost) is not compatible anymore with WordPress 3.5, please DH update Page Speed (apache mod_pagespeed) to final version 1.0. Maybe 1.0 is compatible with WordPress 3.5.

This status page is useless, unless things are going okay! MYSQL down!

wehn i reported this a few weeks ago to support that mysql has been unusable for several weeks now, they diplomatically told me to f**k off

Server is down for us too! If it’s not down, it’s super slow.

My site was fine this morning, and now can’t be reached at all. I submitted a support ticket and have heard nothing back.

HELP!

Our site was back earlier today, but dropped off again about 15 mins ago….Please fix.

FYI – It gives a “Error establishing a database connection” message

All sites down

It would be really nice if there was some information regarding the new east coast server ‘rockbridge’. I was told that there would be 1-2 days of slow loading, then everything should be back to normal. About to be into day 4… What is going on…. How much longer until performance is back to an acceptable level? I have some very unhappy clients

!@#$%^ been down all my wordpress sites getting a crazy 403 error. Dreamhost…. you better fix yourself!!!!!!

Database down, again.

My wordpress is not problem!

Good work!!!!

Barcelona turismo!

thanks for the unscheduled server move (DEC 18th) which has crippled my site one week before Christmas. Two help desk tickets created over 24 hours with no response. Site is 10x slower and no Stamps.com integration (says sql issue)

The upgrade completely obliterted our database. Tried repair and restore but to no avail. We didn’t even sign up for automatic upgrades!

dreamhost mysql is generally unusable. Your wordpress site on DH will take you to back to the nostalgic 1997 modem speeds.

Why do you people even *have* a status page if it’s not being updated? All sites down, not a *word* from DHstatus or the twitter feed for “breaking info”, but lots of twitter people @-ing dhstatus/dhcare with reports and questions. FIX YOUR STUFF. And when you can’t, LET US KNOW IT’S DOWN AND WHY.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail delays with homiemail-sub4 mail server (RESOLVED)
3:45 am

Mail delays with homiemail-sub4 mail server (RESOLVED)

Posted (November 16th, 2012 at 3:45 am PST) by patrickm

We are currently experiencing delays with mail being delivered from our homiemail-sub4 mail cluster. Our team has resolved the root issue which caused the problem, but delays of up to a few hours are being seen still at this time. Email from this cluster will be delivered, but may take more time than usual until the queues on the servers have been cleared out.

You can see if your mail accounts are on this cluster by logging in to your Web Panel and clicking on the “Account Status” drop-down in the upper right section of any page. Under “Your Email Cluster” will show “homiemail-sub4″.

Please feel free to contact Support (from the Support > Contact Support page in your Web Panel) if you have any questions or if you are experiencing any other issues!

Update Nov. 17th 6:43pm PST: The backlog of messages has been taken care of and this is now resolved!

Update Nov. 16th 6:28pm PST: Unfortunately, we are still working to clear out the backlogged e-mail. It’s moving through at a steady rate but please expect that there will still be delays until all backlogged messages are cleared out. We are very sorry for the inconvenience this is causing and appreciate your patience.

Update Nov. 16th 11:15am PST: The backlogged e-mail is still being cleared from the cluster at a steady rate. All email from this cluster will be delivered, but will still be delayed while the backlog is processed.

Update Nov. 16th 9:35am PST: Our admin team has located and fixed the source of the issue, but mail will continue to be delayed while our mail servers process the large backlog of mail. Email from this cluster will be delivered, but may still take more time than usual until the queues on the affected servers have been cleared out.

 

Update Nov. 16th 7:35am PST: We are still working on getting mail processed as quickly as possible, but unfortunately, delays of a few hours or more are still to be expected. Thanks again for your continued patience!

 

Update Nov. 16th 6:25am PST: We’re noticing some of our mail servers are still having a bit of trouble processing mail and are seeing what can be done to speed that along. Thanks again for your patience and we will continue to keep this post updated with further news as we have it.

This entry was posted on Friday, November 16th, 2012 at 3:45 am and is filed under Email Issue, General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

38 Responses to “Mail delays with homiemail-sub4 mail server (RESOLVED)”

Go DH, GO! In DH team we trust!

how long will be solved?

still how many hours will it take time to deliever mail

still have problems, how long it takes to solve 100%?

It seems that “homiemail-sub4″ is also down. I’ve stopped getting emails!

I’m still have problem with delays!
One message was sent by an user at 8:15AM GMT-2h and not received yet (12:05PM GMT -2h).

Best regards,
Renato Yamane

This always happens on important days for me. Please fix this soon.

We are not experiencing delays as I can’t even get past auth at this point? An update would be appreciated as we are now at the 5 hour mark from first report!

DOH – 3 hour mark, my bad!

Outgoing mail seems to be working, but we’ve not received email for hours.

Again.

We still are not receiving emails – do we know when this will be fixed? I know it states emails will be slow but there is a difference between slow and not receiving them at all. We need to start seeing emails.

do you hane an idea when the problem will be solved?

Thanks for looking into this… am glad to see that Dreamhost sees that there’s been an issue, and that some mail servers are already back up. I appreciate this service.

Before this problem, even DH website and panels were not working for 3-4 hours. Our web server and mail server was down. When they’ve go up then we saw that mail problem.

They didn’t mention anything about their website, or panel downtime.

So, we hope they can solve.

Those emails sent in server problem time, can I expect to receive them?

By the way, they went down on 9 hours ago and go up after 3.5 hours. And we’re waiting 5.5 hours for this mail problem to be solved.

Ha ha!

Mail is kinda important you know…just 10-hours/year brings you down to 4-9’s reliability… …still don’t have my mail for the day.

al momento il problema persiste………………

email = work = money … thanks …

What is going on?
I just had several e-mails bounce back to me saying my IP was blacklisted: “Client host [208.113.200.5]”
I tried to make a report with Cloudmark Sender Intelligence™ (CSI), and they say a report has already been filed:
“Remediation Already Requested. A request for remediation has already been made for the IP address: 208.113.200.5. Only one request can be made per 24-hour period.”

Is it due to the backlog of e-mails that are now getting slammed at servers? We need this fixed. Our reputations are on the line here !!

BTW I’m on homiemail-sub5 NOT homiemail-sub4.

Great, was just delaying my sent mail late last night, now they are actually bouncing back undeliverable. Its not like its Friday and Im trying to get an invoice submitted for payment before the weekend. DH always fails at the most inopportune times. FIX this shit.

Great, thanks for the anticipated black friday.

purtroppo il problema continua…………….

Come on guys. I’m losing money!!!!!!!

podrán decir si tenemos novedades, no se puede trabajar así. Qué es lo que pasa¡¡¡

I posted a ticket…didnt realize this was happening..
Go Team Go!

There is any update about this issue?

The e-mails are around 4 hours delayed. I don’t think the issue is solved at all.

Please we need you to find out and solve the problem. It’s desperating to work like this.

Best regards!

still does not work. email forms do not arrive.

still does not work. email forms do not arrive. help

At least 3 business-critical e-mails (of which I am sure they have been sent) dating from November 16 (during the downtime) still didn’t arrive (and probably never will). I am sure that quite some mails are lost forever.

I am also still missing all the emails sent during the server downtime on Friday. This issue is not resolved yet!

This is not resolved. No email sent our web site is being received.

let’s forget about for lost messages, I just want to e-mail server works

I have noticed that mail sent from my shell account using the PHP mail() function is being delayed for several hours on an intermediate server called “pinkiepie”. This may be the same issue a number of you are seeing. I have submitted a support request but have not received a response yet.

THIS IS SITILL NOT WORKING!!!
PLEASE FIX or GO DADDY!!!!

Well, I just received a response. They acknowledge the delay on the intermediate server (a mail relay) and say they are working to bring the queue load down….

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance for US-West Data Centers Saturday November 17th 6pm PST (6 hrs) (RESOLVED)
12:24 pm

Network Maintenance for US-West Data Centers Saturday November 17th 6pm PST (6 hrs) (RESOLVED)

Posted (November 14th, 2012 at 12:24 pm PST) by andrea

Starting Saturday, November 17th at 6pm US Pacific Standard Time, we will be conducting planned maintenance on network circuits that connect our US-West data centers (Los Angeles, Irvine, and LAX) to each other. These circuits provide connectivity for overall network redundancy, and improve performance for customers with Web and MySQL services in multiple US-West data centers. To find out if you may be affected by this maintenance, go to the DreamHost panel main navigation and check under Status > Data Centers, and check if you have associated Web and MySQL services at different US-West data center locations.

The maintenance window is scheduled for 6 hours so that we can perform the upgrades across all of the relevant network systems, then test and confirm that it has been succesful. However any one affected customer (with services in two or more data centers) should not experience any more than up to 10-20 minutes of unavailability of services during the maintenance window, due to the way we’ve planned it out and will implement it to minimize customer impact.

We appreciate your understanding and patience while we conduct this maintenance. It is a necessary part of our Series of Network Improvements, and it’s a key step before we can complete our final phase core router upgrades.

Please note, this maintenance is dependent on the successful completion of the Hardware Replacement being performed on Wednesday and various pre-maintenance sanity checks. Once all of these checks are completed successfully, we will update this status post by Friday noon PST to confirm that it will proceed as planned.

We will also update this status post and open the comments once we start, and will provide periodic updates until conclusion. If you believe you are being affected by an unrelated service issue, please contact Support through the DreamHost Panel. We will also be monitoring Status Post comments throughout the maintenance.

Update: Nov. 17th 7:37pm PST: This maintenance window is now finished, with no downtime! We expected some, but were able to pull everything off with none! Thank you for your patience!

Update: Nov. 17th 7:04pm PST: We’re now routing internal traffic over our new hardware and testing. Again, this shouldn’t cause any downtime as of now.

Update: Nov. 17th 6:40pm PST: We’re still prepping the backend of things, mainly IPv6, so there shouldn’t be a shred of customer facing downtime at this moment. We’ll keep things updated here!

Update: Nov. 17th 5:55pm PST: Our team is getting prepared and will be all set to perform this maintenance. We’ll make sure to open up, and monitor comments in this blog, as well as our normal support channels. We’ll also keep this post updated as things happen.

Update Nov. 16th 12:25pm PST: Everything went according to plan last Wednesday, so this maintenance window is a GO for tomorrow, November 17th at 6pm PST. Again, we’ll open up comments, and monitor them for issues. We appreciate your patience during this update.

This entry was posted on Wednesday, November 14th, 2012 at 12:24 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Core Router Hardware Replacement for US-West Data Center (Los Angeles, CA) Wednesday November 14th, 8pm PST (4 hrs) (RESOLVED)
4:49 pm

Core Router Hardware Replacement for US-West Data Center (Los Angeles, CA) Wednesday November 14th, 8pm PST (4 hrs) (RESOLVED)

Posted (November 13th, 2012 at 4:49 pm PST) by Ralph

Wednesday November 14th, starting at 8pm Pacific Standard Time, we will be replacing hardware on our core router in one of our Los Angeles based data centers. There will be no service interruption during this maintenance, however we are scheduling a 4 hour window where we will be monitoring that the replacement goes smoothly. Once this maintenance starts, we will open the comments and will be watching closely to make sure it is completed successfully.

Update Nov. 14th 9:15pm PST: We’re all done with no downtime! We are closing this maintenance window now, as the internal hardware upgrade went as planned. This allows us to continue with our maintenance window scheduled for next Saturday, at 6pm.

Update Nov. 14th 8:45pm PST: We’re still humming along! Currently, our engineers are bringing up part of the replacement hardware. Everything is going as planned. We’ll continue to monitor for problems.

Update Nov. 14th 8:12pm PST: Ok, we’re starting the first step of this maintenance. Again, this shouldn’t be customer facing so no downtime should result in any changes made here.

Update Nov. 14th 7:52pm PST: Our team is getting prepared and will be all set to perform this maintenance. We’ll make sure to open up, and monitor comments in this blog, as well as our normal support channels. We’ll also keep this post updated as things happen.

This entry was posted on Tuesday, November 13th, 2012 at 4:49 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Core Router Hardware Replacement for US-West Data Center (Los Angeles, CA) Wednesday November 14th, 8pm PST (4 hrs) (RESOLVED)”

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » VPN service to be discontinued
2:44 pm

VPN service to be discontinued

Posted (November 7th, 2012 at 2:44 pm PST) by mir

For some time now, DreamHost has offered a VPN (Virtual Private Network) service to securely connect to DreamHost servers through our partner, CloudConnector. At the point that we introduced it, this service was useful for many of our customers, as most other technologies available at the time (such as Telnet and FTP) did not support encryption. Today, however, encrypted protocols such as SSH and SFTP are widely available, other superior VPN services are readily available, and the software required to connect to our VPN is not compatible with many newer operating systems. As a result, we have made the decision to phase out our VPN service.

If you are one of the approximately 250 customers who had active VPN service, we will be prorating your most recent charges for this service and applying them as a credit to your account — please feel free to contact DreamHost Support if you have any questions.

This entry was posted on Wednesday, November 7th, 2012 at 2:44 pm and is filed under Announcement, DreamHost Features. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

13 Responses to “VPN service to be discontinued”

our emails are down since last night and I can’t get in touch with Dreamhost because your panel site is also partially down, I keep walking in circles trying to contact support. We need help, we are losing business deals because of it, it is a huge loss! somebody please help!

Pretoria server is DOWN

do you know when it is going to be fixed, I am having business problems because of it!!!

Our websites are intermittently down today! Does ne1 know what is going on?!

all sites all down,gosh!

all sites down trying a VPS.

This is a NIGTHMAREHOST.

NIGTHMAREHOST = LOOSING $$$

LAREPUTAMADREQUELOSREMILPARIOHIJOSDEREMILPUTAS

mysql server down…. again…

Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110 [2013]

Order confirmation emails not sending, submitted support ticket with no response from Dreamhost

I thought I was the only one with this problem… Just a few minutes ago, this hit my website and freaked me out!

mysql server down…. again…

Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110 [2013]

is ftp joker down? i cant connect to it.

I’ve been having connectivity issues with the hosting service Dreamhost for the past 5 years and have been unable to continue building a business. Because of the unreliability of Dreamhost I am down to 3 clients, from a previous total of more than 20. I am changing my career, and I would like to thank Dreamhost for giving me the “push” I needed. Website design and development is a thankless business and I’m happy to be leaving it behind. Because of Dreamhost I am truly following my dream, and I am grateful for what they’ve done.

No offense, but if you all are stupid enough to post your complaints after reading “Please note that this is not a way to contact DreamHost.” right above, and on an unrelated status post, perhaps there’s other reasons your supposed businesses are failing…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Intermittent outage in our US West LAX Data Center (RESOLVED)
10:47 am

Intermittent outage in our US West LAX Data Center (RESOLVED)

Posted (November 6th, 2012 at 10:47 am PST) by Oscar

We experienced a brief network outage in our US West LAX Data Center, which caused web and MySQL downtime (email services were not impacted by this issue). Our system administrators identified the cause as a network “hiccup” in the core router which caused connectivity issues within our LAX datacenter. This outage lasted approximately 15 minutes and corrected itself while our Network Engineering team was investigating. We are, however, noticing issues between mysql machines communicating to web servers which are still causing sites not to load correctly. We are continuing to monitor all services and will make sure to update this post if we notice any further issues. We apologize for the trouble and appreciate your patience. Please contact support for any issues you have related to your site being down. You can do so by submitting a ticket through the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.

Update November 6th 2:30pm PST: Our Network Engineers have given the all clear and we are resolving this status post. All DNS queries are successfully resolving and we have not seen any further DNS/mysql issues. Should you have any further issues with your sites, please submit a support request and we will be glad to look into the issue for you. -Oscar

Update November 6th 1:51pm PST: It appears the network problems have been resolved. All DNS queries are successfully resolving. We will leave the status post up. However, it is safe to say that this issue is fixed. Our admins will continue to actively monitor the network. Just to ensure that everything continues to flow smoothly. If you continue to experience problems, please submit a support request and provide the details. -Brian S

Update November 6th 1:18pm PST: As previously mentioned, our admins have made configuration changes that will help to repair / speed up dns resolution. Thus far we are noticing some improvements. I’ve spot checked incoming support and found sites to be working. Access from remote locations are a success as well. We still aren’t marking this issue as resolved, but things are looking promising. We will continue to update this post as we get more information. If you are still having issues with specific domains, please contact support. -Brian S

Update November 6th 12:30pm PST: Our admins have made configuration changes that will help to repair / speed up dns resolution. Thus far we are noticing some improvements, but are NOT calling this issue resolved. We will continue to update this post as we get more information. If you are still having issues with specific domains, please contact support. -Brian S

Update November 6th 12:10pm PST: We have determined that one of the core routers suffered a failure that caused the initial network hiccup. While the router has recovered, the failure resulted in some missing vlan configurations that we are currently trying to troubleshoot. We will continue to update this post as we get more information. If you are still having issues with specific domains, please contact support. -Oscar

Update November 6th 11:50am PST: Our network engineering team is still investigating the DNS and mysql connections issues we are seeing in our LAX Data Center.  If you are still having issues with specific domains, please contact support. -Oscar

Update November 6th 11:30am PST: We are continuing to investigate the DNS and mysql connections issues we are seeing in our LAX Data Center.  If you are still having issues with specific domains, please contact support. -Oscar

Update November 6th 11:10am PST: We are still investigating issues related to the core router “hiccup” which seem to be limited to DNS and some mysql connections.  If you are still having issues with specific domains, please contact support. -Oscar

This entry was posted on Tuesday, November 6th, 2012 at 10:47 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

100 Responses to “Intermittent outage in our US West LAX Data Center (RESOLVED)”

Sux…, everything is down to me… :(

Can’t ssh to alkalbalrai.dreamhost.com.

For the last 30 minutes or so, Dreamhost server(s) seems to be unable to do DNS lookups also. This is going to impact things like Akismet spam checking on a wordpress site or any other services that are outside of the Dreamhost network.

Our site also appears unable to do DNS lookup—a WP plugin to display a Google .cal feed isn’t working, and an attempt to have the server download plugin updates failed. With our site unable to update the calendar feed, a critical part of our website is broke.

> email services were not impacted by this issue
Our email has been down since around 1:30pm Eastern.

My entire website is down.

We’re down too. A political blog down on Election Day, gotta love it!

My website is down too. This is taking a lot of time.

@Charles: What’s the error you are getting on your site? I was able to load newageofpolitics.com without any issues. Please let me know so I can check a few things.

I *did* contact support. They pointed me at this blog.
MySQL is not working for us, bringing our whole site down.

“SQLSTATE[HY000] [2013] Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111″

I’m having problems resolving DNS too,

I can connect to server via SSH and websites is working, but when I try to get something (from ssh terminal)resolving doesn’t work.

I have tried to ping google with ping google.com <- could not resolve
ping 8.8.8.8 (DNS Google) Returns ping okay

I hope that the problem will be resolved soon

thanks..

My site down too. On a day when I would otherwise be getting tons of traffic. Seriously Dreamhost??? Get your act together.

no reply from support, still getting database error message

I have some sites down & my account is very slow..

I’m having troubles in MySQL too. Anxious to get this solved ASAP.

Thanks.

Still timeout to mysql hosts….

Not sure if this is related to the general outage. Is anyone else having problems with PHP scripts suddenly not working? Code that has been working fine for at least a year all of the sudden is locking up this morning. I am using MagpieRSS 3rd party, worked like a charm – until today.

Dreamhost, please don’t refer to these things as “resolved” “hiccups” when they aren’t. You end up with a credibility problem.

Still down with MySQL issues.

I can ping and try to connect, but it answers me “Too many connections”.

Here is a little more info on my RSS problem, getting the following error:

Warning: MagpieRSS: Failed to fetch http://online.wsj.com/xml/rss/3_7011.xml (HTTP Error: connection failed () in /home/orbusmax/orbusmax.com/magpierss/rss_fetch.inc on line 238
Wall Street Journal:

Never have seen this error before, and ALL my rss fetches are returning the same kind of error, meaning it’s not just one site that I can’t connect to for their RSS feed.

Does this sound like it’s related to the DNS/Ping issues other people are talking about?

Still seeing DNS lookup errors on horologium. Load time is at 10.0 on this machine, probably because so many processes are timing out on DNS requests.

My sites is down, and my askimet does,t working :( OMG!!!!

Still waiting for the support reply! More than 4 hours OFFLINE!

I’m receiving an Error Establishing A Database Connection error. Please let me know when this “brief” outage will be fixed.

Our mail server is down to.
I have a application to launch today using smtp =/

One of my sites is down from this, sometimes I get a bit of a page others I get “site down” on cloudflare. On other sites I have things load fine but in admin pages I can’t load things like WP RSS feeds. Hope things get fixed soon.

Ours down to….when will it be resolved?

I’m also having issues that are probably related to DNS. I have some PHP code that is making network requests using cURL and OAuth and neither of them are returning anything, just timing out. Would LOVE to have this resolved ASAP so I can continue working.

Also – general FTP and SSH connections are seeming to take a lot longer than usual. They do eventually connect, but take almost a full minute to do so.

Keep us posted please!

Thanks.

All my MYSQL sites are down as well. Over 15 clients with no website. Going on 4 hours when we discovered it at 10:10 CST.

I feel for the user with the political blog. Can’t imagine that….

I’ve got one of the top targeted congressional sites in the country down right now. Not a good day.
Dreamhost, I see no reason that I’ve now dealt with sites going down at least every other month

I too am having trouble with SQL connection on my site. Thank you for trying to fix this.

Site offline completely. Noticed it at 11:15am.

My sites down too.

These things always seem to hit me, I manage to get my site going and the host has issues. Please, please, please. Get it back up asap :(

Seems like this is officially more than a ‘hiccup’.

I need to know immediately if this is something that will be resolved in a few hours, or if I should spend that time transferring everything over to a new service. Please advise.

Yes, officially more than a hiccup. I wonder what percentage of customer base this is effecting.

My sites are down now as well.

enif server is down.

Come on Oscar! You can do it!

((( all my sites are down as well

Right at payroll…CRAP
Anyone know a good site to learn about rolling your own Mysql site? (I’m actually serious about that)

we have connectivity issues with our databases, when the system will be repaired?
thanks
Antonio

Hiccup is quite the euphemism. My site went down at first, came back after about 20 minutes and is now down again?? What did you mess up the second time?

I need my site back. my website will online this vote poll today. oh my god

i need our website back online. it’s 3..30pm and still down. what can you do to help?

My site just went down about 20 minutes ago. No connection to database. Ugh!

Great. Now our sites aren’t even resolving in DNS!

Any word on how long this is going to take? All my sites are down.

Sites back up for me.

All websites on VPS still down 4pm (EST)

@Jassen: Your sites are now back up and resolving. I had to restart apache on your VPS.

My site is still down due to the database being down. Need to get this fixed. Should I restart my server too?

Our sites are still down!

@Andrew Funderburg

Domains on both of your accounts are loading. If the problems persists, could you please submit a support request

Back alive! Thanks!

FINALLY back up. 2-1/2 hours after noticing it was down.

MySQL still down on our VPS. Going on an hour and a half now.

Finally back up. Thanks.

@Jamie

Could you please provide your domain name?

my sites are all still unreachable due to DNS it looks…

@Jamie

I will look into your issue and respond to your support request

@Dave

Your site loads fine for me. Please try clearing your browser cache. If the problems persists, please submit a support request.

My incoming email has not been happening for 3+ hours now. Everything has been checked. Clearly email was affected as well. Waiting for a reply to my support ticket…

MySQL still down on our VPS.
Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

On then off on then off. Still having issues.

Its friday 7 at 9:00 AM an all websites are SLOW , please solve it NOW!

Things are getting worse, please DO SOMETHING!!!! All sites down again

We have problems receiving messages from the contact form. The service is intermittent since yesterday.

Database Error: Unable to connect to the database:Could not connect to MySQL AND ALL MY SITES!

Nope issue isn’t resolved until all your customer sites are backup.

Getting “Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.” on Google Chrome when I try to visit my sites. And “The connection was reset” on Firefox.

All websites down, email not working – dreamhost panel seems down as well. What’s going on? I’m in trouble…

We are still having problems with the email programs like thunderbird, outlook express or microsoft outlook are experiencing errors.
There is a turnaround in some machines are working, I checked the settings and they are equal.

Control Panel is out for me, as also all my sites @ alcor server. I can only read emails via web servers, outlook and thunderbird are experiencing errors.

This problem, whatever it is, is NOT RESOLVED.

While my sites and the panel are apparently “up”, or accessible, to some people, they are not accessible for me, and apparently other people.

Panel is not accessible, the server the sites are on is not accessible, webmail is not accessible.

November 8th, 2012 at 6:51 amPablo Salaverry Says:

I have the same problem. Some people can see some sites and I can’t. I’m having serious problems about this issue. The support told me that all is fine but it is not. Can you please check this and give an answer please?

Did you email the support? Since I can’t access Panel, I emailed them with example traceroutes to various sites to see if they can identify the problem.

Some of my sites load and some don’t, yet I’ve asked other people, and they can access the the ones I can’t, and http://www.isup.me says they are up, and traceroute and ping from different sites seems to work.

But there is DEFINITELY a problem. And I’m not the only one who has it.

And for those people who think their sites are working okay, because you can access them: be aware that myself and maybe many others will NOT be able to see your sites, even though they are loading for you.

In other words, even though you think there is no problem because your sites are loading for you, there IS a problem, and you probably are losing traffic without even knowing it.

I am not receiving any mail through Apple Mail which will no longer connect to the server. I’ve been told by Dreamhost support that its my settings and a problem with Apple but these have not been touched since the work on the DH server.

Can only connect through DH webmail.

Hi,

Our sites have been down for at least the last 10 hours and we cannot access the panel. Please acknowledge the problem and get it sorted!

November 8th, 2012 at 8:58 amPablo Salaverry Says:

Jim, I have exactly the same problem. The funny thing is I can see a site and no other and both are in the same server with the same user. I can’t access to the ftp either from those sites.

I wrote several times to support and they told me that I have to clear the browser’s history cache… They dont have any clue about this problem. I don’t know what to do now :(

DAN: ironically, I can’t get to webmail but can (fortunately) download mail via my email software. No reply yet from DH support though.

PABLO: same here, regarding two sites with same user, where one loads and one doesn’t. It’s complete BS to say it’s browser cache. I’ve tried on two entirely different computers, with different operating systems.

http://www.site1.com – won’t load
http://www.site2.com – does load

Same user, same server. Thing is it’s possible to log into the server using site2.com.

Ie:

ssh sameuser@site1.com (the one that doesn’t load) = doesn’t work, times out
ssh sameuser@site2.com (the one that does load) = logs in fine

Same server, same user. One works, one doesn’t, in web and SHH.

I don’t use FTP, but are you saying the site that loads won’t let you log in via FTP (ie, if you use the domain of the working site)?

I know what I’m going to do if DH don’t start admitting and fixing this problem, instead of trying to pretend there’s nothing wrong: find another host.

Great, my comment is “awaiting moderation”. No idea how long that will take, so here it is again (with example sites removed):

DAN: ironically, I can’t get to webmail but can (fortunately) download mail via my email software. No reply yet from DH support though.

PABLO: same here, regarding two sites with same user, where one loads and one doesn’t. It’s complete BS to say it’s browser cache. I’ve tried on two entirely different computers, with different operating systems.

[site1] – won’t load
[site2] – does load

Same user, same server. Thing is it’s possible to log into the server using [site2].

Ie:

ssh sameuser@[site1] (the one that doesn’t load) = doesn’t work, times out
ssh sameuser@[site2] (the one that does load) = logs in fine

Same server, same user. One works, one doesn’t, in web and SSH.

I don’t use FTP, but are you saying the site that loads won’t let you log in via FTP (ie, if you use the domain of the working site)?

I know what I’m going to do if DH don’t start admitting and fixing this problem, instead of trying to pretend there’s nothing wrong: find another host.

Check your sites.

Looks like stuff that wasn’t working for me (at least) is starting to work again. Hopefully permanently.

Hopefully yours is working too now.

November 8th, 2012 at 10:04 amPablo Salaverry Says:

Jim, my sites are working now, but yestarday I have the same issue. So I hope this is a permanently solution too.

Mine just started working in the last 20 minutes, after posting my 9:45am post. Good to hear yours is working too, sounds like we had the same or similar problem.

Oh, and thanks for posting, even though it’s never good to have a problem, it’s helpful to know that other people have a problem too.

down again…..

We are experiencing web outages at thepeachpress.com. We just launched this site and are in a critical marketing period. Please help getting it back up.

Site is still down.

Downhost.

Our site is down as of now, it has been for at least 12 hours now. Please help.

This is getting ridiculous. Site has been extremely slow/down for hours and hours

Site apparently is back up, but extremely slow. Please Dreamhost help us out!

November 13.

All sites down again.

Hi,

my sites are down

site is still up and down.. whats going on DH… october was an awesome month, december has gone to shit.. check my pingdom downtime graph:

http://i.imgur.com/6N9ii.png

thats 5.5 hours downtime almost just for november, and it’s only just half way through the month!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Core Router Software Update for US-West Data Center (Irvine, CA) Sunday November 11th 6pm PST (4 hrs) (RESOLVED)
8:50 pm

Core Router Software Update for US-West Data Center (Irvine, CA) Sunday November 11th 6pm PST (4 hrs) (RESOLVED)

Posted (November 5th, 2012 at 8:50 pm PST) by Seohee

Sunday November 11th, starting at 6pm Pacific Standard Time, we will be applying a manual software update on the core router of our US-West Data Center located in Irvine, CA. Customers with services in this data center will experience less than 20 minutes of downtime when the software update is being performed. You can find out what services you have at this data center by checking the Status->Data Centers section of the DreamHost Web Panel. This maintenance affects the following services:

Web, E-Mail, MySQL, DreamObjects, Dedicated, VPS

We thank you for your understanding and patience during this core router software maintenance.

We will be updating this status post right before the maintenance, then provide periodic updates through to the conclusion of the maintenance. We will be actively monitoring support submitted through the DreamHost Panel as well as comments posted here.

Update Nov. 11th 7:20pm PST: We are calling this maintenance completely resolved now. The software update has completed successfully and the issues with the final networks is resolved. We thank you for your patience and communication throughout the maintenance.

Update Nov. 11th 7:05pm PST: We have all but one network restored and working correctly now. We’re almost done!

Update Nov. 11th 6:48pm PST: We are still investigating the reason why these remaining networks did not come up correctly. We are working our provider and working on the issue ourselves, more details to come.

Update Nov. 11th 6:35pm PST: We are still cleaning up some final routing issues. We expect to have them resolved shortly. The majority of sites are up without issue. We also noticed comments were off and those are now turned on so you can report any issues you may be having.

Update Nov. 11th 6:20pm PST: Traffic is now restored and we are now spot checking to make sure everything is running smoothly.

Update Nov. 11th 6:12pm PST: Services are down now due to the software update currently running.

Update Nov. 11th 6:05pm PST: We are starting the software update now. Service will be affected for anyone with services in the US-West Data Center in Irvine, CA. We expect service interruption to be less than 20 minutes.

Update Nov. 11th 5:55pm PST: Our team is in place and we are getting ready to start this maintenance. We will be monitoring support and the comments on this status post. Any updates will be reported here as we progress through this update.

This entry was posted on Monday, November 5th, 2012 at 8:50 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

49 Responses to “Core Router Software Update for US-West Data Center (Irvine, CA) Sunday November 11th 6pm PST (4 hrs) (RESOLVED)”

I realize this isn’t an officialy way to contact Dreamhost, but at the current time the web panel is down and so are my websites…

Whats happening!? 4 hours ago… was using site now and suddenly stopped.

Website down…panel down.. server down..

Same here. Everything is down. Same thing last night.

… and so is dreamhost.com

Panel’s down too – good to know it’s not just me

I’m supposing it may be a bad sign you have complete outages just 12 hours later after a network update. So, yea, “good job”.

Everything is dead for me too. Panel and ALL sites

I suppose it’s a testament to your skills that even your own damn site is down, not to speak of everything else. Wouldn’t you at least think to build some redundancy? Dreamhost has been slowly building towards this crash what with all the problems we have been having in the last few months.

Site and MYSQL server are down. DH web-panel down and DH DNS servers seem to be down too. what’s up?

“I’m supposing it may be a bad sign you have complete outages just 12 hours later after a network update. So, yea, “good job”.”

Ummm, check your calendar. The maintenance you are referring to isn’t scheduled to occur until the UPCOMING week-end.

Shoot first, aim later.

Everything is down! Even the Dreamhost Panel! EMERGENCY!!

Same here. All my sites and Panel are down, clients keep calling, this is great!

Site outages are happening WAY too often with DH these days

Yes, EVERYTHING is down right now–even http://www.dreamhost.com !

c’mon dreamhost, get the panel, your site and my site and my email back up!

Could we please get an update on when your problems are going to be solved?

Totally unacceptable!! I have clients waiting for material.

Yeah…everything is down…not a good thing to say the least…

DREAMHOST…PLEASE POST LATEST STATUS UPDATE POST-HASTE :-(

This is really bad, specially this happened 2 mins after we tweeted a link to about 1000 followers… Thanks! Brilliant!

Greets Dreamhost team I have issues with all my sites and web proyects. Could you confirm what’s going on with the server and if you have plans ti solve this in a quick way?

Thanks

OK, everything is mysteriously working once again.

And, we’re back…

Having problems all day. First my databases went down, then one time only dreamhoststatus.com, other time the panel, and now panel, dreamhoststatus and even dreamhost.com. Seems to be working again now. Hopefully it was the last time.

Yes we are back. But everyday I’m having MySQL server downtimes about 15-30 minutes. This is NOT OK.

Should I use file based database ??????

Aggelo, same for me. My site went viral, so I thought that’s a problem because of that. Turns out, it’s not.

somewhat seems back, panel doesnt seem to load properly

Still down here in Brazil. What’s going wrong DREAMHOST? Can you put my sites back?

They are up, but really slow, comparing to any other time.

Database down. Please fix.

Is anyone else having problems with PHP scripts suddenly not working? Code that has been working fine for at least a year all of the sudden is locking up this morning.

There is a new status post related to the outage that is affecting our US West LAX Data Center. Please make sure to post there if you are having any issues with your sites:

http://www.dreamhoststatus.com/2012/11/06/intermittent-outage-in-one-of-our-los-angeles-data-centers-2/

We are still seeing issues with mysql machines communicating to web servers which is causing site downtime. We will continue to update the status post there.

I miss the service I received some time ago…
Maybe is time to move to another better hosting.

MySQL service is still down for me.

my site and all services are down please address

We are experiencing web outages at thepeachpress.com. We just launched this site and are in a critical marketing period. Please help getting it back up.

If posting about service interruptions, please let us know the affected domain(s) so that we can spot check them. Thanks! -Dan W, DreamHost Support

All of our sites are still down.

My Site is down. Please address

massive.ph
mercs.ph
warinc.ph

Thank you for the examples everybody, we have confirmed that there is still a handful of servers without connectivity and are working to restore service to them ASAP. We will update this post shortly.

Website and db are still down

Some site and mysql down in my panel,such as http://www.gzboai.org ,can you visit it?

ziqijin.com is still down. maybe it’s because of static IP?

Both gzboai.org and ziqijin.com are working correctly for me.

@Chris @Ziqi @Mary, I tested all of your sites and they are coming up fine both internally and externally for us now.

Yeah it’s up now. Thanks guys

All is well here.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Maintenance for US-West Data Centers Saturday November 3rd 8pm PDT (6 hrs) – (RESOLVED)
1:19 pm

Network Maintenance for US-West Data Centers Saturday November 3rd 8pm PDT (6 hrs) – (RESOLVED)

Posted (October 30th, 2012 at 1:19 pm PST) by andrea

Starting Saturday, November 3rd at 8pm US Pacific Daylight Time, we will be conducting planned maintenance on network circuits that connect our US-West data centers (Los Angeles, Irvine, and LAX) to each other. These circuits provide connectivity for overall network redundancy, and improve performance for customers with Web and MySQL services in multiple US-West data centers. To find out if you may be affected by this maintenance, go to the DreamHost panel main navigation and check under Status > Data Centers, and check if you have associated Web and MySQL services at different US-West data center locations.

The maintenance window is scheduled for 6 hours so that we can perform the upgrades across all of the relevant network systems, then test and confirm that it has been succesful. However any one affected customer (with services in two or more data centers) should not experience any more than up to 10-20 minutes of unavailability of services during the maintenance window, due to the way we’ve planned it out and will implement it to minimize customer impact.

We appreciate your understanding and patience while we conduct this maintenance. It is a necessary part of our Series of Network Improvements, and it’s a key step before we can complete our final phase core router upgrades.

We will update this status post once we start and provide periodic updates until conclusion. If you believe you are being affected by an unrelated service issue, please contact Support through the DreamHost Panel. We will also be monitoring Status Post comments throughout the maintenance.

Update November 19th 11:20am PDT: This maintenance has been conducted and completed without downtime on 11/17th, so it’s considered resolved. Please see more details at Network Maintenance for US-West Data Centers Saturday November 17th 6pm PST (6 hrs) (RESOLVED)

Update November 3rd 12:15pm PDT: Tonight’s maintenance has been postponed. We will repost once we are ready to commit to a new date for this maintenance to occur.

This entry was posted on Tuesday, October 30th, 2012 at 1:19 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » US-East Region Storm Preparedness (RESOLVED)
4:33 pm

US-East Region Storm Preparedness (RESOLVED)

Posted (October 29th, 2012 at 4:33 pm PST) by andrea

As many of you who are based in the US are aware, there is a major storm system impacting the East Coast today, with potential impact to Virginia where our US-East region data center is located. This notification is to advise that DreamHost is well prepared to handle any power outages or other issues that may arise during the storm and its aftermath.

At this time, we have very few customers provisioned in our US-East region data center, because it was only brought online in August 2012. So the vast majority of customers will be entirely unaffected by any issues related to the storm. You can find out where your services are located by logging into the Panel and clicking “Status” under the main menu on the left, then “Data Centers”.

For those few customers who have services in US-East (Ashburn, Virginia), we want you to know that we do not expect any service issues or outages during the storm or its aftermath, because we have layers of redundancy in place for all major systems, as follows:

• US-East is on two entirely separate power grids, so a failure of one grid will not impact services;

• if both power grids go down, we have sophisticated battery and generator backup systems (uninterruptible power supplies) in place that will keep the data center online for 3 days before the generators require fueling, and indefinitely with generator fueling arrangements in place;

• there are generator-refueling contracts in place with a number of suppliers to ensure we can run the data center indefinitely on generators;

• our network connectivity to US-East is protected and redundant; and

• the building is rated for major storms and will not be impacted.

We will update this status post if any issues occur at US-East, but we do not expect any during this storm.

Update: Wednesday October 31sth 5:30pm PDT: We are happy to report that we’re in the clear. We were not affected at all, and it’s no longer necessary to continue monitoring the situation. We are resolving this post, but feel free to contact support if you have any questions.

Update: Tuesday October 30th 11:40am PDT: The brunt of the storm has passed, and all operations continue to be normal. Even though surrounding areas were and are affected, our US-East (Virginia) data center maintained utility power throughout the night. We do not anticipate any further cause for alarm, however we are continuing to closely monitor the situation.

This entry was posted on Monday, October 29th, 2012 at 4:33 pm and is filed under Announcement. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Proactive Security Maintenance (New Login Keys)
11:01 am

Proactive Security Maintenance (New Login Keys)

Posted (October 24th, 2012 at 11:01 am PST) by mir

This is just a heads-up for everyone that you will likely see a key change warning the next time that you are logging in with your shell user. This is not a cause for concern as it is the result of a security maintenance which we are performing to prevent exploitation of weak or outdated keys (the maintenance should completed in the next 24 hours). The resulting warning will look like this (but will show your user and the server key which we have removed from the example):

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
REMOVED FOR SECURITY REASONS
Please contact your system administrator.
Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message.
Offending key in /home/USER/.ssh/known_hosts:10278
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

To get rid of the warning, there are three methods. Please keep in mind that the modification to fix this needs to be performed on the client side, so if you are getting this connecting from home/work, it is that computer that needs to be updated.

The first method, you can delete your entire known_hosts file if you have several hosts in need of updating. On first login it will confirm the new key.

The second method, you could run in your shell a command like “ssh-keygen -R HOSTNAME”.

The third method, the known_hosts file will need to be edited. In the above case, one would open the file /home/USER/.ssh/known_hosts in their preferred text editor, and remove the problem line.

The error message lists the line number; it can be found after the file name in the error. In the above example the line number would be 10,278. If you are using the “vi” editor, you could type “:10278″ without quotes to get to that line, then hit the “d” key twice to delete it. Other editors should have similar features to assist you in removing the line of text.

(If you have accessed your server from any other computers before this key change, the same change will need to be done there on your next ssh or sftp login to the server.)

That should cover the situation but please let our support team know if you have any questions or concerns and thank you for choosing DreamHost!

 

Update: Tuesday October 30th 1:33pm PDT:

 

We’ve got a brand-spanking new panel tab now available, which can be used to look up the correct keys for all machines on your account! You can access this by clicking “Users” > “SSH Keys”, or by following this link!

This entry was posted on Wednesday, October 24th, 2012 at 11:01 am and is filed under Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

60 Responses to “Proactive Security Maintenance (New Login Keys)”

Thanks for posting this. I was wondering if I should open a ticket. Glad I checked here first.

Thanks from me too…I got REALLY worried when the host key warning popped up.

How do we know that the presented “wrong” fingerprint is in fact correct, and not a well-timed man-in-the-middle attack?

How do you remove the old key? Like this:

ssh-keygen -R your_hostname_here

October 24th, 2012 at 12:01 pmmasterwebmonkey Says:

“How do we know that the presented “wrong” fingerprint is in fact correct, and not a well-timed man-in-the-middle attack?”
You don’t unless your hosting company tells you about it… I am not worried now that I saw this.

So I will need to update every one of my users in every one of my machines, good one!!

First, I love DH services, and I am very grateful they take proactive measures for security purposes. Full SSH shell access is one of the top reasons I use DH hosting. That said:

A) This is a shining example of DH pulling the rug from beneath our feet with no warning. This will crash any automated SSH scripts.

B) Where does DH publish their (true and real, signed with a known public key) server fingerprints, so that we can actually verify we are not subject to man-in-the-middle attacks?

C) Please also publish the public keys and fingerprints for your webmail servers. Your instructions to simply use HTTPS and ignore any pesky warnings is dumb.

If you’re not SSH savvy, note that the listed path (/home/USER/.ssh/known_hosts) is on your local machine. The number after the file name “known_hosts:10278″ is the line number (each line contains a known host entry). See Ed’s comment above for removing the old host entry. You can also just delete this file, but it will then require re-confirmation the next time you connect to any previously known host.

And I’m wondering what is more secure, if someone knowing that DH updated all SSH keys try to do a man in the middle attack to a dreamhost customer, the customer will think that is normal because of the update, Just saying…

I contacted Support before reading this after receiving the SSH notice. I asked to verify the new key, but they just directed me to this page for information without verifying the key! How can we be sure it’s the true key if they won’t tell us that it’s correct? Any reason why we can’t check the authenticity of the key in the panel? Seems like a trivial bit of code to add to the status box.

Eduardo – excellent analysis.

Here’s another failure case due to this update:

Warning: the RSA host key for ‘yourhost.com’ differs from the key for the IP address ‘1.2.3.4’

This implies that your known_host file contains multiple hosts with the same IP address (multiple domains/users on the same host at DH), which now have conflicting keys.

I think you need to remove all known_hosts entries with the server’s old key. I just deleted my known_hosts file, then connected to each of my DH accounts to confirm the fingerprint (which could very well be a fraudulent man-in-the-middle fingerprint unless you actually verify it from a known trusted source, as reported by Eduardo above).

DH sure does keep you on your toes!

@Ed: That is another way to do it, thanks. It was added to the post for additional info.

@neb: I do apologize for not more notice being provided. I will bring up for discussion your other two points with our security and admin team.

@bob: We will consider adding this to the panel, but at this time it isn’t available there. You may contact support to verify the key you are seeing and we can provide additional confirmation.

Sloppy and unprofessional. Bob’s comment irks me even more. Even a mass email yesterday would give your customers more assurance and time to prepare. Calling it “proactive” is a joke.

I have never heard of host keys needing to change as any kind of routine security measure, let alone without warning. How do we even know this announcement is legit?

@Axl: There are various reasons that they would need to be changed, mostly related to making the systems more secure. The keys will appear to change to users who are moved to a new server, whether it is shared, webserver vps or dedicated, as the physical server hosting a site changes. The announcement is legit, and if you need confirmation you may contact DreamHost support via the control panel.

Why, oh, why must I always search out information when you make changes? How hard would it be for you to copy and paste your blog entry into an email message and mass mail it to your clients? You have an easy enough time with sending out your newsletter. Surely someone on your staff has the skills to send an email to customers before you do things like this. Warnings and notices that you are about to do something are far better than explanations and apologies after.

Oh, and FYI, those instructions of yours only help if someone is in a shell or on a *nix box. I just want to upload a file from my Windows machine so I have to go find the answer myself. Thanks for ensuring I won’t meet today’s deadline.

If you have a bunch of domains on Dreamhost, here’s what I’d suggest to do instead if you’re on a unix system (e.g. Mac, Linux)

Run the following in your terminal
ssh-keyscan -t rsa yourdomain.com

It’ll return the RSA key, e.g.

hd-trailers.net SSH-2.0-OpenSSH_5.1p1 Debian-5
hd-trailers.net ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAlZabh2ylFQDwtVO+B2Ff8KQVp6TQD8xxmpaPQAeHZ3wYuCL0LB2uqXLbKn73AjqSKqbW198CW+eRLf6V/fvjQCdiXeRaE+kbK3ZrCN9YTTL8rIbRx+sCDAUgUSKUslUkg+h4Vo6ppaPrkcWPlQKYnbJ3GMw9pRGKkOKug3fAP2DS4MUugMxoGbpcych2CPxOyQPdBOR3lMAUcDd0cB56Ehq10NST/NgwaHCP+7uh+pUHepeqpxW3fqa5NGtX8FNMzN1oN4nZTbmrKqWa30dRufiQL8YKYH6QFbY6aj9lvh74TcdOsKDAFjPX9AHNVZdcrJ2Gl03HUtO4S4qrlbMSBw==

Open ~/.ssh/known_hosts in your favorite text editor
Search and replace the old RSA key with the new one.

Tada! (^_^x)

Thanks for posting this. I’m glad I checked here!

Thanks for this last tip about ssh-keyscan. What I am not totally clear on, and I have to explain it to my tech-averse board, is: We trust this because? All it really tells me is what is being broadcast by whomever is actually serving the domain at the moment. But if dreamhost were down, hijacked, well, how would this key, generated this way, help me if there were a man-in-the-middle attack.

I’m pretty darned worried, actually. I am accustomed to machine fingerprint codes being public knowledge and easily accessible, like md5sums on an iso download. Am I expecting too much? Dreamhost has rebuffed my every effort to get this routine information, and, really, it’s my one complaint with them. I am ready to pull my domains on the strength of it.

Peace,


properclinic.com
unclog courts

Glad to see the notice here, but worried that I could’ve easily missed it. Notices like this should probably be sent in email, too, so that we can know to update our automated ssh scripts. For example, I’ve got an rsync cronjob to my Personal Backup space. Don’t we all? And presumably, they all just broke.

This is fine for techie SSH users, but I am using Dreamweaver, and while I am familiar with RSA keys, I am not that savvy about any of the things the instructions are talking about.

Any help for me?

From the post:

“The fingerprint for the RSA key sent by the remote host is
REMOVED FOR SECURITY REASONS”

Okay, so am I way off base, or is the bit that was removed the ONE IMPORTANT PART OF THE POST.

Robert Link is right, these key fingerprints need to be public knowledge so that a person at any given time can be sure who they’re connecting to.

You guys at Dreamhost might need to look at how PKI works, and removing a public key fingerprint from a post like this “for security reasons” doesn’t make you guys look like you have sysadmins that know what they’re doing.

Spankminister: Thanks for the kind word. If you are near enough Cal-Tech might you be interested in a key-signing meetup? My contact info is upthread. rl

@Justin, I’ve contacted support THREE times asking them directly whether the new key is correct for my server, and each time I just get a response saying that the keys have been changed and direct me to this post. I’ve been asking for SSH keys via the panel for about two years. It’s a pain to have to confirm the keys every time i’m moved to a new server or, in this case, the keys are changed.

Hey guys, you forgot to post the list of new keys…

https://twitter.com/DreamHostCare/status/261215166492979200 says “the key change was kind of a fix asap and release issue.” Was this indeed proactive or as a result of a security issue?

Dreamhost probably has hundreds if not thousands of SSH keys (one for each server). But still, the key fingerprint(s) need to be published somewhere, either on a website signed with a PGP key, or in the panel.

The old keys were probably generated with a vulnerability, and when they realized it, they fixed it.

Dreamhost technical support emailed me (after three rounds of pointing me to this page). The new ssh host key is:

b0:aa:77:f9:2c:c0:7b:c1:0a:c9:87:19:64:73:1d:1b

as per what my ssh client was telling me,
so I’m confident I’m talking to the right host.

–Benji

Just update your keys and quit whining, heathens.

@Steve (and other Dreamweaver users) – you’ll want to trash (or edit) Dreamweaver’s ssh_hosts file. It’s location can vary based on your OS and version of DW. On my Mac, it was at:

/user/Library/Application Support/Adobe/Dreamweaver CS5/en_US/Configuration/ssh_hosts

And thanks, Dreamhost, for the ‘proactive’ communication on this.

Thanks, @Michael.

I didn’t see your response, before I tried this, which deleted the preferences and config files. That worked.

http://helpx.adobe.com/dreamweaver/kb/restore-preferences-dreamweaver-cs4-cs5.html

Many of us have valid security concerns here, and it’s an oversight on the part of Dreamhost for not being proactive about handling those security concerns from the users’ perspectives (rather than from a Dreamhost security administrator’s perspective). I am in China, and I have been subject to man-in-the-middle attacks. For example, at one time I was getting different fingerprints at different times and I actually needed to get on the phone with a friend in the US to verify I was receiving the right public key. So, let me be clear: This is not whining—I need to have strong assurances about any public key that I’m going to accept.

Now, I absolutely need to know if Dreamhost uses the same keys for all its servers. The fingerprint being shown by my ssh client is not the same as what Benji has posted above.

Lastly, Dreamhost, please know that an untrustworthy and/or difficult security system is a deal-breaker for me. Please be truly proactive by responding publicly and openly about the raised security concerns so that we can all rest easy about Dreamhost security.

I’m using rsync to connect to Dreamhost via ssh in a cron job.

I deleted the bad line in known_hosts as directed. The next time cron ran the script I get:

Host key verification failed.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]

Was I supposed to put in a new key or something?

@Al: when debugging an application over ssh, it’s best to start at the lowest level and work your way up. So start by sshing in directly to your Dreamhost server. Once you get that working (I would *assume* you’ll have to provide some input, perhaps just a simple ‘y’es), try to rsync some files manually. *Then* once you have gotten ssh and rsync working, let the cron job try again. At least, that’s the script that’s always worked for me.

Generally: I’m getting a different key fingerprint than what @Benji has posted (I’m seeing a2:59:f8:11:b7:ad:9c:01:ad:d5:b7:ea:75:0d:ab:50; I used to connect to ruchba, has that changed?). Has anyone come up with a way to verify that’s the right fingerprint?

Finally, the *really* easy way to remove an obsolete key from your known_hosts file is with sed. If it’s in line 10278 (BTW I’m *really* impressed by that number. I ssh around a lot but I don’t think I’ve ever hit even 300 lines in my known_hosts, let alone 10k!!), then try:

sed -i 10278d ~/.ssh/known_hosts

You can prove for yourself it will do what you expect, first, with an experiment like this:

$ for f in abc def ghi; do echo $f >> test.txt; done
$ cat test.txt
abc
def
ghi
$ sed -i 2d test.txt
$ cat test.txt
abc
ghi
$

I have to agree with what others are saying about the “REMOVED FOR SECURITY REASONS” line. This just shows complete and total ignorance of the Public Key Infrastructure. The point of a public key is for it to be… wait for it… publicly available! It should be published far and wide. There should be an easy and secure way to verify its authenticity. While being published on a non-secure website such as dreamhoststatus.com would not fulfill the “secure” part of that, it would be better than nothing! I personally would recommend both putting the SSH Key on this post AND providing a link to an https://www.dreamhost.com page that also contains this information… at the very least, do the later.

Not only does “concealing*” this information not help improve security, it actually actively HINDERS security. In order for it to be secure, people need to know with 100% certainty that they are connecting to legitimate servers. I really would have hoped that the security personnel at DreamHost would already be aware of this :(. After this post (both the suddeness of it and the ignorance displayed in it) I’m not sure that continuing to stay with DreamHost is the right move.

* I use the term “concealing” loosely. Not showing this fingerprint in this post does not keep it from view. It’s exceedingly easy to simply attempt an ssh connection to a server to retrieve that server’s public key. Of course, this is by design as the public key, as already stated numerous times, is SUPPOSED TO BE PUBLIC. (I’m hoping that if I say it enough times, DreamHost will get a clue.)

Please publish the new key so that we can verify it since you are obviously changing it for some unscheduled, security related reason.

Thanks for breaking everything. Now I have to reset all the crons, contact vendors that were using the keys and update various other apps.

On Kubuntu/Ubuntu I find the following helpful removal note to easily remove the offending keys:

Offending RSA key in /home/mistry/.ssh/known_hosts:2
remove with: ssh-keygen -f “/home/USER/.ssh/known_hosts” -R SERVER

So much simpler if your server is “my_website.com” , and your user is “you” , just run:
ssh-keygen -f “/home/you/.ssh/known_hosts” -R my_website.com

You will likely need to have openssh-client package installed to have such helpful utilities:
sudo apt-get install openssh-client

@drewbenn Thanks for the info. I did what you said… ssh’d into Dream from my server (which I ssh’d into from my local iMac) and Dream happily wrote its key to my known_hosts file. I logged out of Dream and ran my cron script again which uses rysnc to transfer a few files each hour from my PairNetworks server to some space I have on Dream and it worked just fine.

I don’t know much about the key stuff as I set it up years ago and have had no problem. Is the key that Dream writes to the known_hosts file some kind of big secure secret? Can you or anyone explain why they just didn’t publish the long key string and tell me to: “Delete the one in your known-hosts file an pop this new on in.” ?

Anyway, thanks for getting us back up and running. The Dream customer service emailed some kind of fingerprint code but never told me what to do with it. Your instructions to log in via ssh and answer ‘yes’ to the code (which matched what they had send me via email) did the trick.

Why did they make this so much harder than it had to be?

Any idea why rsync inside the cron (bash) script would not write the new key to known_hosts but manual ssh did.

Al

Please publish your new SSH host key(s).

Preferably in an PGP-signed message with a well-signed PGP key.

Thank you.

I’ll add another to the round of “Where’s the list of keys?”

Also, the more use of PGP (signing blog posts, emails, ssh public keys, the webserver CA certificate, etc.) the better. Those who don’t care will overlook the signatures, but those who do care can verify the authenticity of communications.

Even though the initial dissemination of a master dreamhost public key may not be verifiable, at least that key can gain some trust over time. Much like I came to trust the ssh key over time.

From the twat linked above, https://twitter.com/DreamHostCare/status/261215166492979200 it sounds like someone figured out how to read the private keys from the hosts. Any more info on this?

@All: If in doubt about the key that your server should have, please don’t hesitate to contact support and we can provide the fingerprint for you. Again our apologies for the issues this has caused.

Just saying someone owes me a clean pair of pants.

This is why I love Dream Host! Thank you guys for keeping us up to date!!!

Used 3rd method no issues. Thanks for detailed writeup.

@Al:

> and it worked just fine

Neat! :)

> Is the key that Dream writes to the known_hosts file some kind of big secure secret?

No. Anyone can get the key just by typing “ssh (servername).dreamhost.com,” and DreamHost’s server sends the key right away by plaintext. That’s by design, and doesn’t affect the security in any way. That’s just how the server identifies itself, along with or before (I’m not terribly clear on the actual mechanism) the magic happens behind the scenes to ensure the security of the communication.

I wanted to verify this was true, so I asked at http://security.stackexchange.com/questions/23129/ .

> Can you or anyone explain why they just didn’t publish the long key string

Apparently not. They *should* have done that (or at least, published (in an authenticated fashion) the fingerprints), which would have made a lot of the commenters here happier.

> and tell me to: “Delete the one in your known-hosts file an pop this new on in.” ?

The error message you get when you try to ssh in does the first part: it warns you there is a key mismatch (what you have recorded doesn’t match what was just sent), and you can take action as a result of that, if you wish.

Once you’ve deleted the old key, if you connect again and accept the new key, ssh will push (not pop — haha, CompSci joke) the new key to your authorized_keys file for you.

> The Dream customer service emailed some kind of fingerprint code but never told me what to do with it.

Ideally, verify that the fingerprint is what your ssh client reported to you when you tried to ssh in to the server. Or verify that fingerprint against what’s stored in your authorized_keys file, like this (which, BTW, creates a temporary file called ‘temporary.txt'; you can delete that when you’re done. For some difficult-to-follow information on what those commands do, try typing the command “man ssh-keygen”):
ssh-keygen -F yourhostname.com -f ~/.ssh/known_hosts > temporary.txt
ssh-keygen -l -f temporary.txt

> Your instructions to log in via ssh and answer ‘yes’ to the code (which matched what they had send me via email) did the trick. Any idea why rsync inside the cron (bash) script would not write the new key to known_hosts but manual ssh did.

The cron job expects to see certain things. It saw, and logged, the “Are you sure you want to continue connecting (yes/no)?” prompt — but it wasn’t expecting to have to enter any information there, so it probably just kept entering the commands from the rest of the script (or, maybe, just hung because it was waiting until it saw a prompt it expected). Meanwhile, the ssh connection never finished connecting because the your ssh client never got the “yes” it wanted to get. Once you logged in manually and *saved* the server’s fingerprint to your known_hosts file, your ssh client stopped needing to ask you for permission to connect: since the next time you tried to connect, it recognized the remote server as one that you had already approved as being legitimate. So your cron job worked like it used to, because ssh (behind the scenes, from the cron job’s perspective) established the connection with no problems.

> Why did they make this so much harder than it had to be?

This is *exactly* the way things are supposed to work. Well, preferably with some advance notice. And when someone change their keypairs, they need to publish the public key (and/or the fingerprint) in an authenticated manner (which DreamHost didn’t do, and still hasn’t done). Not that publishing that information would have helped *you*, just some of the more paranoid commenters here (like me). But all the stuff you went through is normal, and expected. Changing keys like this is a big deal (and happens rarely: it might be 10 years before this happens to you again), since the keys are what your computer uses to verify that it’s talking to the correct server (the analog would be someone you only talk to over the phone changing their voice: suddenly you can’t identify them anymore without some additional information). When the keys (what the server uses to identify itself) change, your computer (effectively, in your case, the cron job) needs *you* to verify that your computer is still talking to the correct server.

This is exactly the sort of situation that could have been completely avoided by using Monkeysphere. It adds PGP on top of SSH keys so that you can automatically add new keys and revoke old ones whenever you want without anyone having to touch any file or any script breaking.

If you’re going to publish those keys and PGP-sign them like you should have done long ago, might as well do it in a machine-readable way, mhm?

How frigging hard would it be to publish the signature of the key here, so we can verify that the key being offered is indeed correct? Does Dreamhost even *KNOW* how these things work?

I may or may not have established a well-timed man-in-the-middle attack on one of Dreamhost’s servers. The fingerprint for the public RSA key I may or may not be using in said possible attack is 81:85:24:ba:39:5f:0e:e0:aa:64:bd:bc:fb:b1:d4:be.

Now, if you see that fingerprint when trying to establish an ssh connection with Dreamhost, the only way you’ll know for sure whether that fingerprint comes from a Dreamhost machine or one of mine is if Dreamhost securely confirms (e.g., through a signed and/or encrypted email to all customers) what the true fingerprints are.

Please, Dreamhost, I implore you to demonstrate that you take our security concerns seriously. Either disseminate the fingerprints or provide a clear explanation as to why you can’t/won’t.

I’m sorry, but this is bad. I am a simple user, using Fetch (on a Mac) for sftp connections. Fetch simply says that it is not possible to connect. Nothing more. And to be honest, this happens frequently. So after waiting a while I realized that something else was happening.

But after finding this “status” update I really do not understand what is said here.

In my Finder there is no .ssh directory. There is no known_hosts file anywhere to be found. Probably it all hidden. But I don’t care. As a simple user you are asking to much from me. I really think this is not the way to make changes.
And, as a simple user, I have no clue how to do this better :-) Hoping someone will help you (Dreamhost) with that.

It is a hidden file on your Mac that can, usually, be found at:

/Users/YOURUSERNAME/.ssh/known_hosts:1

Assuming that you are only using Fetch to connect, do this:

* In Fetch, choose Fetch Help from the Help menu.
* In the help window, search for “known hosts”.
* Double-click on the “RSA host key differs warning” help topic (it should be the one at the top of the list).
* A help topic opens in the help window.
* Scroll down to the “Delete the known_hosts file for me” link.
* Click the link, and follow the instructions.

That is assuming you are using a version of Fetch before 5.7, if you are using a newer version, then the “RSA host key differs” help topic has been removed from Fetch Help because Fetch now (as of Fetch 5.7) automatically handles this situation. If you connect to a server and there’s a host key mismatch, you should see a dialog explaining the situation, and asking you how you would like to proceed.

HTH!

@Wondering: thanks a lot for this clear explanation. I’m using Fetch 5.3. Maybe time to upgrade. Always afraid to break something that works well.
The way to get rid of the problem is probably different for every program. Makes it hard for Dreamhost to help us all.
i still cannot see the hidden files. Shall look for some help around here.
Thanks again!

October 28th, 2012 at 6:50 pmJames Alexander Says:

FYI, my site is on the “fir” dreamhost server, and the RSA finger print I got when connecting is as follows:

12:cb:b3:1f:e1:8e:2c:cf:0e:b7:cd:7c:66:32:84:76

Wow Dreamhost finally fixed the flaw I found, could you please update my 3 month old support ticket?

Yes, private keys were easily accessible.

Thank you @Wondering and @m1st0 finally able to connect via SFTP.

I like DreamHost in many ways, but this is one of my gripes with them. I’ve previously attempted to find a solution. See here: http://wiki.dreamhost.com/Talk:Security

sampablokuper: We’ve just added a panel tab (Users > SSH Keys) to address this issue. It’ll list the proper RSA and DSA key fingerprints for all servers accessible by your account, as well as a known_hosts fragment with full public keys for you to import.

https://panel.dreamhost.com/index.cgi?tree=users.sshkeys

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Core Router Software Upgrade – US – West Data Center (Los Angeles, CA) Sunday October 21st 8pm PDT (4 hours) POSTPONED
4:28 pm

Core Router Software Upgrade – US – West Data Center (Los Angeles, CA) Sunday October 21st 8pm PDT (4 hours) POSTPONED

Posted (October 19th, 2012 at 4:28 pm PST) by Ralph

This Sunday, October 21st at 8pm PDT, we will be performing a software upgrade in our US West Data Center based in Los Angeles, CA, in a 4 hour maintenance window. This upgrade was already successfully applied to our US West – Irvine, CA Data Center on October 3rd. The successful upgrade did not effect customer’s services, so we are confident this current upgrade will come with similar results. There are redundant routing engines in place which will allow us to minimize any service interrupting impact. Each engine can be upgraded individually in order to keep service active during the maintenance. We will update this status post once the maintenance starts and provide periodic updates until conclusion.

You can now view your service data center locations in the DreamHost Web Panel under Status->Data Centers.

Update: Sunday October 21st 11:10pm PDT: The rollback has completed, and all systems are up and running. After we’ve had a chance to analyze logs and data we’ve retrieved during this process, we will reschedule the core router software upgrade for another time. We will announce in a new status post as soon as possible. Thanks, once again, for your patience!

Update: Sunday October 21st 10:35pm PDT: As it turns out, unfortunately, we aren’t currently able to complete the upgrade without causing downtime. As a result, we will be rolling back the upgrade and rescheduling this upgrade for another time; we will keep you posted. More updates to come as the rollback occurs. Thanks for your patience!

Update: Sunday October 21st 10:15pm PDT: To make sure everything continues as expected, we’ve opted to switch to a manual upgrade. We are working on this now, and will continue posting updates as the process continues.

Update: Sunday October 21st 9:30pm PDT: We continue working on upgrading our backup routing engines, and are also in contact with our vendor.

Update: Sunday October 21st 9:00pm PDT: We are continuing to monitor the upgrade as it runs. As expected, no customer-facing services have been affected. More updates to come as we have them.

Update: Sunday October 21st 8:30pm PDT: The upgrade process has begun, and we are currently in the process of rebooting our backup routing engines.

Update: Sunday October 21st 8:00pm PDT: We are getting things ready to begin the software upgrade and will be updating this post with updates as progress is made. Stay tuned for more updates.

This entry was posted on Friday, October 19th, 2012 at 4:28 pm and is filed under Network Maintenance, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Core Router Software Upgrade – US – West Data Center (Los Angeles, CA) Sunday October 21st 8pm PDT (4 hours) POSTPONED”

does this affect a VPS server or access to MySQL services? – using WP and am getting a sever error (server down for maintenance.)

No, this was not customer-facing; also, it does not appear that your servers are in the data centers where the maintenance is occurring (as you can see in your panel as well via the link above). Your site does seem to be up from what I can see, but please be sure to contact support under Support / Contact Support in your panel if you need any further assistance!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrade to Roundcube Version Monday October 8th 8pm PDT (1 hr) (RESOLVED)
7:58 pm

Upgrade to Roundcube Version Monday October 8th 8pm PDT (1 hr) (RESOLVED)

Posted (October 8th, 2012 at 7:58 pm PST) by Dan J

We will be performing a software upgrade to the Roundcube installation to version 0.8.2. We estimate the upgrade to take around one hour. During this maintenance customers may experience intermittent outages or loss of session. We will of course keep you apprised of our progress as we work our way through the maintenance window.

Update: Monday October 8th 8:59pm PDT: The upgrade is now complete. If you notice any problems with the new version then please contact support through the Dreamhost panel. Thanks for your patience.

Update: Monday October 8th 8:00pm PDT: The upgrade is beginning now. We will post more details as they are available.

This entry was posted on Monday, October 8th, 2012 at 7:58 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Core Router Software Upgrades (West Coast Data Centers) Thursday October 4th 8pm PDT (6 hrs) POSTPONED
12:14 pm

Core Router Software Upgrades (West Coast Data Centers) Thursday October 4th 8pm PDT (6 hrs) POSTPONED

Posted (October 3rd, 2012 at 12:14 pm PST) by andrea

We will be performing a software upgrade which was recently made available on Thursday, October 4th starting at 8pm PDT. The window will last for 6 hours and will allow us enough time to upgrade both of our West Coast Data Centers.  This software upgrade is necessary to complete prior to the switch over to our new Core Router system.  We do not anticipate any service interruption during the upgrade as there are redundant routing engines in place. We are able to upgrade our non-active engine and then bring it online to take over while the others can be upgraded while they are inactive. We will of course keep you apprised of our progress as we work our way through the maintenance window.

 

Update: Thursday October 11th 4:50m PDT: To clarify, the router upgrade in one of the data centers was successfully completed. The other one is scheduled to happen soon. We’ll add a new status post as soon as we’re ready to go. As always, thank you for your continued patience while we’re improving our network infrastructure.

 

 

Update: Friday October 5th 1:40am PDT: The rollback has completed and all services have remained up and stable throughout the process. This concludes our maintenance for today; we will post another update once we have rescheduled the intended upgrade. Thanks, once again, for your patience!

Update: Friday October 5th 1:17am PDT: We are moving onto the last component needing to be rolled back. We will let you know once this has completed.

Update: Friday October 5th 12:38am PDT: The process of rolling back the upgrades as mentioned in the previous update is continuing; this may require a reboot once the downgrade completes which may cause a few minutes of downtime. We will continue to keep you updated throughout the process. Thanks for your patience!

Update: Thursday October 4th 11:23pm PDT: The upgrade process has hit a snag on two of the backup routing engines, and we’ve deemed a roll-back of the upgrade to be the best course of action at this time. We’ve begun rolling back the upgrades and will update this post with further information as it becomes available.

Update: Thursday October 4th 10:30pm PDT: The second batch of upgrades is ongoing. We will keep you updated on the progress as we get closer to completion.

Update: Thursday October 4th 9:59pm PDT: The second group of upgrades is processing at the moment.

Update: Thursday October 4th 9:36pm PDT: The network systems team has now completed testing the first group of updates. Everything appears to be working as planned. The second group of upgrades is about to begin.

Update: Thursday October 4th 8:53pm PDT: The first batch of upgrades appears to be complete now. Our admin team is running tests to make sure everything is working properly.

Update: Thursday October 4th 8:36pm PDT: The first batch of upgrades is nearly complete. So far no problems. We will keep you updated on the progress.

Update: Thursday October 4th 8:08pm PDT: The first upgrade is starting now.

Update: Thursday October 4th 8:00pm PDT: We’re getting ready to begin this maintenance now.

This entry was posted on Wednesday, October 3rd, 2012 at 12:14 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Core Router Software Upgrades (West Coast Data Centers) Thursday October 4th 8pm PDT (6 hrs) POSTPONED”

well my sites are ALLL DOWN…

Miguel’s Account (#343091)

My site is down …

My PS went down at a very inopportune time. It would be great if you sent out an e-blast or something for planned things like this so me and my business can plan accordingly.

My site was effected. It’s unbelievable that you guys don’t notify your customers about scheduled maintenance. When GoDaddy plans on doing work that effects my installations, I get a notification usually about a week ahead of time. Why can’t you do the same?

Mine is back online, but I agree with the posters above that an email notification would be nice.

Anyone else experiencing major network slowdown?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade Completed
3:18 pm

Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade Completed

Posted (September 27th, 2012 at 3:18 pm PST) by andrea

We are happy to report that we have completed the installation of an additional 10 gigabit internet feed in one of our our west-coast datacenters. This will further improve redundancy on our existing 10 gigabit lines. There was no negative impact during the installation, as the primary network was not affected. You should be noticing improved speeds on some of your services, so we wanted to share the good news.

Thank you for your continued patience while we further improve our network infrastructure.

This entry was posted on Thursday, September 27th, 2012 at 3:18 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade Completed”

Thanks for your continued service and support.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade – September 20th, 11am PDT (2 hrs) (POSTPONED)
4:10 pm

Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade – September 20th, 11am PDT (2 hrs) (POSTPONED)

Posted (September 17th, 2012 at 4:10 pm PST) by andrea

This Thursday, September 20th, we will be installing an additional 10 gigabit internet feed in one of our our west-coast datacenters. This will further improve redundancy on our existing 10 gigabit lines. We are not anticipating any negative impact during the installation, as the primary network is not affected. After the install, you should be noticing improved speeds on some of your services.

Thank you for your continued patience while we further improve our network infrastructure.

Update: Monday September 24th 10:50am PDT:  Apologies for the lack of updates. This maintenance has been postponed, and we’ll post the new information as soon as we have it.

This entry was posted on Monday, September 17th, 2012 at 4:10 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

26 Responses to “Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade – September 20th, 11am PDT (2 hrs) (POSTPONED)”

Can we please get an ETA when the ‘alaraph’ server move will be completed? My website has been down pretty much all day, and I still have no idea when it will be up. I know I know, if I want guaranteed uptime, I need to find another solution… but this kind of stuff seems to be occurring more and more and it’s costing me money. Any there never seems to ever be a warning ahead of time. And why can’t server moves and other network admin stuff be done at night when most businesses don’t have a lot of traffic and minimize the impact of downtime? This is getting very frustrating.

@Jim – they don’t read these very often. If you have Twitter, you can follow them & interact with them there much better than here.

An extra 10 gigabit feed… Oh, I wish I had one of those at my house!

Yeah maybe an email for people that host on these servers would do? It’s tough doing this at 11 in the middle of the day. That is when most of us have peak traffic.

Thanks Dreamhost!

From an e-mail and my web panel, all my sites are still down/not restored:

“This is an update to let you know that the restore is now stable on your server, and your sites should be loading as expected. However, if you need assistance with missing or outdated files and you don’t have local backups, or if your sites aren’t performing as expected other than being somewhat slow or timing out occasionally due to the high restore load, please contact us with the specifics. You can reach us by either replying to the original announcement message from your inbox or by submitting a ticket through the DreamHost panel, (https://panel.dreamhost.com/) under Support/Contact Support.

We will update this post again later today unless there is a change in status which necessitates additional notification. ”

I contacted support Sep 18, 2012 – 09:54:43, no response.

I love the heads-up on this. I have people complaining that my site is not loading. *sigh*

Mi sitio anda lento, se cae el administrador. Necesito que todo esté perfectamente online. ¿Pueden explicarme que sucede?

I’m getting MySQL errors and my site is broken.

And now it works, ahh the power of prayer and DH combined, unstoppable :D

24 hours later, my sites are all still down. Got this message 18 hours ago:

“I’m very sorry that you’re still experiencing issues due to the hardware problems on your web-server. Our admin team is aware of and looking into your specific issue. I’ll be back in touch when I receive updated
information from them.

Thanks for your patience in this matter, we will do all that we can to get this taken care of for you ASAP.”

Wish I was having hardware problems, but I’m not. All of my sites’ files are still missing.

How about you send emails to customers regarding your downtime, instead of having us constantly check this site for updates!? SO FRUSTRATING!!!!

There should be some form of compensation for those affected.

Another 24 hours and all my sites are still down. My clients are not happy and neither am I.

Hello: All my MySQL databases (on Bass server) show the message: Too many connections. It is related with this maitenance????

I’m fed up with Dreamhost. Been a customer since 2005, too much downtime.

Getting an error #2003 – The server is not responding.
Submitted support ticket.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

September 21st, 2012 at 1:22 amPatrick Rosenheim Says:

i’m getting 302 errors

me also, trying to log into wp admin and ftp – for wp admin:

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@cleodubois.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

(note, am not listed officially as site administrator but am the client’s designer)

My sites are back up. I never got a status update from support. Not impressed at all, but happy my sites are finally back up.

My email is down in Coronado since yesterday…anyone else having trouble logging into their email?

I have been having issues with email as well. IMAP server failures and sessions expiring within a minute or two.

jaga.dreamhost.com can not visite, any worker attention to this?

knifetooth mysql server has been giving storage errors (error 28) for the last hour and it hasn’t even been reported yet?? Where are you guys?

We have had extremely slow e-mail since Friday along with the lack of ability to connect with our website and blog.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail System Relocation Maintenance Sunday September 16th, 12am PDT (RESOLVED)
1:51 pm

Mail System Relocation Maintenance Sunday September 16th, 12am PDT (RESOLVED)

Posted (September 13th, 2012 at 1:51 pm PST) by Ralph

On September 16th, at midnight (00:00:00), the mail servers in our LAX Datacenter will be transferred to our Los Angeles Datacenter. This relocation will also upgrade our mail service to our freshly hardened and rebuilt network. Since this is hardware relocation, we will be taking mail services offline at approximately 12am Sunday September 16th PDT. We plan to have service restored by 2am PDT and will use the remaining window to make sure that everything is working as expected. During the move, you will not be able to send or receive email. Any incoming mail should be queue’d on the senders’ server and delivered later. However, please note that delivery behavior is dependent on the sending servers configuration, so some mail may bounce back to the outside sender. We thank you for your patience during this relocation and apologize for the inconvenience it may cause. This status post will be updated 15 minutes prior to start of the maintenance, and we will provide you with periodic updates as we progress through the maintenance window.

Update: Sunday September 16th 8:15am PDT: Our admins were able to find an issue with the new cluster quickly and have resolved the issue causing login errors. We’ve done some quick spot checking of accounts and they all seem to be working correctly now. We will now be going through support requests send in regarding this issue and will be doing further testing on those messages. If you are still having problems with your mail, please contact support so that we can look into your issue right away.

Update: Sunday September 16th 8:05am PDT: Our admins are investigating the login errors that cropped up after we resolved the migration post. We don’t have details yet, but we are doing everything we can to resolve these problems.

Update: Sunday September 16th 7:35am PDT: We’re getting reports that mail is having issues and our admins are back on the case investigating it. We’ll update this post soon with our findings.

Update: Sunday September 16th 6:35am PDT: We’ve done extensive testing and have verified that all of the moved mail services are now functioning at full capacity in the new datacenter. All mail services should be working 100% at this time. We thank you for your patience as we completed this necessary system upgrade. If you have any problems with your mail service at this time, please contact support and we’ll be happy to investigate this further for you.

Update: Sunday September 16th 6:25am PDT: The MX issues have been resolved now as well. For the most part, mail should be back up and running now. There are a couple of outstanding issues that may affect a small group of customers still, but our admins are continuing to go through the moved mail services to make sure that everything is working correctly. We’ll update this post shortly with more information, or with the ‘all clear’ if we feel all the issues have been resolved.

Update: Sunday September 16th 6:15am PDT: The mail filers are fully up and running. POP and IMAP services should be back up and running at this time. We’re still working out a couple of MX issues that is holding up delivery of mail. We’re working on fixing that up now to get mail flowing into mailboxes again. More information to come shortly.

Update: Sunday September 16th 5:45am PDT: The mail filers are almost all fully up and mounted. We have a couple more checks to run before fully turning mail back on for the cluster. Stay tuned for updates.

Update: Sunday September 16th 5:15am PDT: The mail filers are in place and we’re in the process of bringing things back online. We don’t have an ETA on when things will be fully back to normal yet, but we’ll update this status post shortly with more information.

Update: Sunday September 16th 4:30am PDT: We’re still working on getting the mail filer machines setup and connected. Once that’s done we should be able to start turning things on and checking to make sure that things are working properly. We’ll update this post shortly with more information as soon as we have it.

Update: Sunday September 16th 3:40am PDT: We have all mail machines and load balancers racked up. We’re working on getting the mail filers all setup and connected now. Unfortunately, we’re going to extend beyond the 4am window we had scheduled, but our admins roughly estimate about an hour for them to finish up. We’ll update this post as soon as we have any more detailed information.

Update: Sunday September 16th 3:15am PDT: The mail servers are all in the datacenter now and we’re in the process of moving them into the server racks and plugging them in. We don’t have an ETA yet on when we expect everything to be back to normal, but we will update this post shortly with more information as soon as we have it.

Update: Sunday September 16th 2:30am PDT: The hardware is on the road and should be arriving at the datacenter shortly. We’ll update this post again once things are unloaded and we start to set the servers up in the waiting racks, which are already wired up and ready to go.

Update: Sunday September 16th 1:30am PDT: It looks like we’re going to be down for a little longer than the 2 hour window that we anticipated. The vans are almost completely loaded and will be leaving for the other datacenter soon. It will be a 30-40 minute car ride and then we’ll need around an hour of setup time after that. We’ll update this post again with more information when we have it.

Update: Sunday September 16th 1:00am PDT: The mail servers are almost fully loaded into the vans that will take them to their new home across town to the Los Angeles datacenter. We’ll update this post again once they arrive and are being unloaded.

Update: Sunday September 16th 12:05am PDT: Mail servers are being shut off now and we’ve already started migrating DNS. We’ll update this post shortly when we have more information on how the migration is going.

Update: Saturday September 15th 11:50pm PDT: Mail services will be turned off in 5 minutes.

Update: Saturday September 15th 11:45pm PDT: We’re getting ready to begin this maintenance now. We’ll be shutting down mail services in the next 10 minutes or so. We expect mail service to be down for approximately two hours. We will post more details as they are available.

This entry was posted on Thursday, September 13th, 2012 at 1:51 pm and is filed under Data Center Maintenance, Email Issue, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

57 Responses to “Mail System Relocation Maintenance Sunday September 16th, 12am PDT (RESOLVED)”

Ok. Waiting for restore.
mail service shutdown now

Dreamhost, how about mentioning important issues like scheduled maintenance in your silly monthly newsletters rather wasting our time with stupid jokes about food cubes?

If you’re thinking about hosting at Dreamhost, ask them first about whether they’re interested in being open about their frequent unannounced service outages first.

Notice would have been helpful for those of us overseas in the middle of a working day!

Yes, posting notice here a few days in advance is *USELESS* as far as giving paying customers advance notice since we normally only come here *AFTER* a downed server is found :(

Sending an email to our account in advance would be MUCH MORE USEFUL.
Thanks

Guys, this is really not the way you should have communicated about this. If you are aware of an upcoming mail outage, please tell us in advance!

I can accept that we in Europe have to deal with outages during daytime (if we wouldn’t want that, we could opt for a European hosting provider), but the least you could have done is update is about this through the proper channels (a mail message, for instance).

My guess is that you know that I am not talking nonsense here, and that you simply chose to not send out that email – or that you forgot about this. In both cases: not a good thing to do. Slap on the wrist for you guys!

I can not access my hosting emails still?, when will this be fixed please etc…

If you are mad because your email isn’t working. You should be. But not at Dream Host, at yourself. You should be prepared for this sort of thing. Dream host a is a great company! They do their best to work for minimal downtime and they do it like a sir.

My mail server works. ;) I host with Dream Host.
You should figure out how not to have any downtime yourself.

Some warning in Advance would have been appreciated! We are in the middle of a site update here in Australia… Not very impressed with DreamHost at all.

11:00 AM in Europe. mail server down. We need the service running as soon as possible for work, thanks.

No disrespect, but can you imagine a mass mailing to all customers? How many people pay attention to the monthly newsletter, or have it turned off, or blocked? Would you have noticed an email from the site? I think, yes, there could have been a better way to receive alerts, but maybe there should be an opt in for them as well. Not saying, don’t email customers when there is down time, but not everyones servers are affected at the same time either, from what I can tell.

So while I think its imperative a move like this needs to be notified to customers, maybe there should be an option on the Panel. Dreamhost.com site to “subscribe” to events like this, where it can be an automatic thing for those who want the alerts, and an opt-in, vs mass spamming every customer about a few hour window of downtime on a Sunday, midnight hardware move that was posted 3 days ago on this site. Sure, we don’t all monitor this site on a daily basis, but I think instead of yelling, we need some sort of constructive suggestions on how to better handle this next time around.

Hindsight is always 20/20, but maybe we should work with them to find a solution to problems, instead of expecting them to generate gigs of traffic for what amounts to Sunday maintenance window, that most companies have every night around the same time of night. For example, I used to work at a bank. Every night, we updated the system. This was to process all transactions for the day, so the next morning, your deposits and withdraws reflect against your account. During the night, there would be several times that you would not be able to access your account or get your ATM balance. This is commonplace with a lot of companies, but I’m using my own personal work experience to illustrate a point here. We used to get customer complaints all the time, that they would be at a casino and couldn’t withdraw money or check their balance or use the debit card. In most cases, Visa does what they call a float, and the queue up the transactions and then post the next morning. This also is a double edged sword, since many customers could take out more money than they had available, and in return, would get hit with fees afterwards for overdrawing their accounts. However, it is common practice that companies have spans of down time every night. Dreamhost for the most part, is down only when there are specific hardware failures or software crashes that effect only specific legs of customers.

In this instance, this is pretty much effecting everyone, but I think given the rarity of instances like this, and “hello”, do we not know systems go down from time to time, they posted this 3 days in advance of the maintenance schedule, choosing the least offensive time and day of the week to make the move. So while I think there could be better communication on events like this, and we should have an opt-in for alerts to moves and change controls like this so we can work around outages, some people just have no concept of what it means to run a system that requires maintenance or daily cycle updates that go down every day at the same time. Just be thankful web hosting isn’t the banking industry, or you would be down on a daily basis, every night.

Not cool, +1 for poor communication and forgetting about your European customers being hit at work time (yes we do work on weekends too). Silly newsletters are cute for some but service outages have to be communicated as an extraordinary event even on the chance of being perceived as spamming by some customers who don’t care.

I second the request for emails before planned outages. Send out an email a day in advance so I can prepare.

This American with insomnia would have also appreciated some warning about this.

Some warning would have been nice, bad timing for customers in Australia. If you sent out emails about stuff like this to paying customers instead of pointless spam we stopped reading long ago it would be a help. I just upgraded my account and this is the second problem I’ve encountered in a week, first one was my website went down now this. Should I be looking elsewhere?

No problem DREAMHOST. I LOVE YOUR Service so I can wait for better datacenter !
Thank You, I love to Pay for Your services :)

Yup, agreed, European here trying to work in the middle of the day – searching for status problems was the only way I heard about this. Billing, communications, your service is really slipping these days.

WTF? (What’s That Feature) Oh, *scheduled*, uninformed downtime you say? WTF indeed.

So let’s join the crowd here : Scheduled maintenance downtimes are only good when announced to the users. (Really !)

When not wanting to do it in the Monthly Newsletter (which I do scan through for system announcements, by the way), at least set up a distribution/subscription list that actually warns those who want to be warned.

As a Datacenter Change Manager for a living and looking at your support page, this is where I would pick my expertise level as “Not to be rude, but I probably know more about this than you!”

Gotta agree with previous posters…we need warning of things like this…clearly it was planned for a good time before it happened.

Why You couldn’t inform us couple of days ago so we can change our websites contacting emails until you get your issue fixed?
You trying to do it in late time thinking it’s smart thing to do, but guess what?
It’s 1:30 PM here and all the service is down for the whole day where I am here.

It would have been nice to receive some warning about this via email. You used to send out announcements about this sort of thing. I’m never one to complain, but it makes better sense to return to a policy of open communication with your bread and butter providers.

What a cock up!

Also wanted to point out, subscribe to the RSS feed, you would have seen this coming: http://www.dreamhoststatus.com/feed/atom/

What the hack…? For me this is not midnight, but in the middle of the afternoon (Denmark). I am in the middle of a promotion campaign and can’t send, answer and even worse, can’t receive mails. Why was this not anounced earlier. I love Dreamhost, great price, great people, but the next time I am gone.

Didn’t see that coming. Any notice upfront ?

This is such a piss take, what the hell are you pricks up to this time. Had no end of problems with servers being down email not working etc, and we pay for this. No warnings either. Hardly professional guys. Need to wake up and sort this crap out and stop dicking around

DigiP, they just sent us out a newsletter a few days ago. Scheduled maintenance items like moving mail servers are things that reasonably should be mentioned in there, not information about chewable, 2 inch hexahedrons.

Europe 14:30
same situation
no email service.

If you’re thinking about becoming a Dreamhost customer, I suggest asking them first about whether they’re going to tell you in advance about any scheduled service outages. If they don’t answer you, look for another hosting company.

Scheduled maintenance is only scheduled if they tell their customers about it first. They don’t. If they did, I might forgive them but their lack of transparency combined with frequent non-scheduled outages doesn’t make it likely that I’m going to renew my contract with them.

Dreamhost got a lot of flak in the past and they created this service blog to try to make their operations more transparent. It’s an improvement, but considering that they don’t even trust themselves enough to put their status blog on their own servers is telling.

No mail for 5.5 hours alreday. I hope you guys are really close to going through this.

Please do something about these updates: almost there, just an hour more, etc. Dont create any expectations when you have no clue how long it is going to take (as it clearly showes you have no clue). What you could update about is how you are going to make this up.

Dear Dreamhost,

Please communicate before any major update. We are in middle east and this is happening in the peak hours here.
Clients are constantly chasing me on this. Please keep communicating on this from now.

Regards,
Hari

This should have been totally seamless, no excuse, this is poor as hell. Get to work and fix this problem.

It’s an entirely sensible decision to put their status pages on a different server if you actually stop and think about it for a minute, but I echo everything else people here have said. But why does this require significant downtime at all? Surely you can ‘just’ set up mirrored facilities in the new location and switch to them or am I being very naive?

Is there a link anywhere to the actual XML of the RSS feed so I can import it into Thunderbird? Damned if I could find one.

I for one DO read the newsletters and the emails that I get from Dreamhost, so it’s kinda unfortunate that they didn’t ATTEMPT to use that means of communication AT ALL. Regardless of whether they think we read them or not they should have tried to use it. Not good.

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

I guess we’re wasting our time here :-(

DigiP, I don’t LIKE RSS feeds. Nor do I like Twitter. I have enough going on in my life without the clucking clutter of too much textual blather.

What, a 2-hour planned outage to physically move hardware? Now up to six? Are you really that unrealistic and clueless? Just exactly what business do you believe to be in?! Intolerable incompetence.

Not impressed of the lack of communication.
As a European customer this is daytime

Less rambling in newsletter – more relevant info thx

Hi dreamhose, can you pleasssse update and let us know an a more exact time for recovery? It’s the weekdays for those of us in the Middle east, so this is literally my monday with email server down and as a freelance designer, nothing is more terrifying.

Also, will my emails be recovered that were sent during this transfer? (fingers crossed you say yes)….

email is up on my servers. No email to speak of coming in, but it’s connecting swiftly.

my email is up too…just don’t know if my emails that were being sent while the server was being transferred where bounced back to the sender or will eventually show up. An entire weekday of no mail could have some negative ramifications on me…but need to know so I can do some damage control :-/

Flowing now. ‘Bout time. Thanks DH.

I’m receiving email now but can not send out?? Will this be resolved in the next few hours?
Diane

Email sending appears to be back up now (11:15am EST). Thanks DH.

I would have appreciated a “Service Notification” (short & sweet) email in advanced – indicating that email server may be disrupted. It would have giving me time to send a courtesy email to my customers indicating the same. I know things get posted but a direct email in this case might have been a good idea.

Hey guys and gals
Sounds like you have had a tough day, well done for getting it all sorted.
I appreciate all the stuff you are doing at the moment with upgrading systems, and client side stuff like 2 factor authentication is appreciated. In general you have been pretty honest and transparent with all your recent problems,and are to be applauded for that.But I too would have appreciated an email beforehand about this sort of hardware move around.
Enjoy your beers tonight.
Mike
ps I like the quirky newsletter:-)

My email or website are still not working.. This is frustrating when 99% of all of my business is email and website use. I need this restored ASAP this is not making me feel to great about your service.. This seems to be standard these days. I am not HAPPY

I believe this issue may not be completely resolved. Applications from a form on our site which we receive via email are not being received.

I work as an Linux IT SysAdmin. This outage while I’d have preferred to not see the hard drives being driven across town think this was a splendid move. Everyone who is complaining about outages being announced forget two things, one, that this site is here (they announced 3 days ahead, this *was* planned), two, that this blog has a RSS feed. you want updates about this kind of thing get the rss feed set up.

The reason why they don’t send out thousands of emails about this is because even doing that once would probably get them blacklisted on the more popular email providers like gmail, msn, and yahoo. So that would cause other issues.

Personally I was confused when I saw my email was down. I went to the panel and saw that DH was doing mail server updates. Knowing how much of a PITA it is to maintain a small setup for mail servers, managing one where it handles thousands of emails a minute.

However on the flip side I think Dreamhost could do better in their time estimation. 2 hours would be reasonable if everything went perfect (it never does). I’d appreciate a more accurate time instead of a hopeful one.

I don’t think the newsletter may be the right place for these kind of maintenance windows. Every hosting company I know of does not send out mass mailings about these because of possible blacklisting/greylisting.

Hosting the blog on a different server also helps in the unlikely event of network outage where the servers are still up but the network is undergoing maintenance that may be outside of Dreamhost’s control.

September 16th, 2012 at 10:33 amBarbara Stafford Says:

Not fixed. All form mail going out from our website also not working… Any idea when it will be fixed?

Does no one use Twitter? They announced this a few days before on their Twitter feed (@dhstatus).

Stoney, what you’ve written makes no sense. Putting scheduled maintenance in the monthly newsletter will not blacklist or greylist them. If that was the case, the current monthly newsletter that we receive would already be blacklisted or greylisted.

Justin, do you read every tweet that you receive? Twitter is not a reliable way to send information to someone who’s not specifically looking for that information. E-mail is.

If they were to say on the status blog, please check Twitter for more on this outage as it happens, that would work for those who checked to the status blog, but as the status blog already has the same information, it would be redundant.

It’s Monday 10:35am pacific and my email isn’t working? Is it because of the Dream Host mail maintenance?

never mind. we are back up. thanks D HOST!

Aquos: It was also announced a few days ago on this very webpage (Dreamhost Status). So you could get the information by: 1) Twitter 2) Status webpage 3) RSS feed.

I don’t think emailing all customers about a scheduled maintenance that most customers won’t even notice (overnight on a Sunday night for US-based customers.. and mail will just queue not bounce) is necessary.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.4.2 Maintenance and Security Release (RESOLVED)
2:42 pm

WordPress 3.4.2 Maintenance and Security Release (RESOLVED)

Posted (September 7th, 2012 at 2:42 pm PST) by mir

The latest-and-greatest WordPress release, 3.4.2 is available for customers to manually install or upgrade from the DreamHost Panel.

You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

The update will be automatically installed for customers who have automatic upgrades enabled. You can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

For more information on the release, check out the announcement post and additional details on the codex.

Known issue:
Custom Field ‘update’ button fails to work properly this can be fixed through using the official Hotfix Plugin.

Fun Fact: WordPress 3.4.2 was tested and live on the DreamHost panel for customers within 15 minutes of release at WordPress.org

This entry was posted on Friday, September 7th, 2012 at 2:42 pm and is filed under DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

11 Responses to “WordPress 3.4.2 Maintenance and Security Release (RESOLVED)”

Uh…I have three sites hosted with DreamHost that are WordPress and now ALL THREE are offline. “Database Error” on all three. What’s going on? HELP

Its because dreamhost does not know how to use MYSQL and thus it dies rather frequently.

-Former DreamHost Customer.
-Happy Arvixe User!

Also Arvixe is about half the cost if you use their personal class. ( Never have had ANY downtime and its really fast.

I have same experience as other dreamhost customer MySQL never is working! Down all the time! DreamHost is suck! I choose ShoveHost website hosting is much expensive but fast and not down. Midphase good too!

At Least your are not stuck on HostGator (being Endurance International now… :( used to be good….. try JaguarPC

My databases are down as well.

DreamHost MySQL down? Must be a Wednesday.

Where is dreamhoststatus.com hosted? I would like to try their service.

Ugh. All my MySQL databases are down and no mention of it on here or on Twitter. No idea when things will be back up either. This is getting a little silly, guys.

dreamhoststatus.com is hosted on linenode.com they are great… expensive but good.

Dreamhost (bootes server) is down right now?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Intermittent outage in one of our Los Angeles data centers (RESOLVED)
1:15 pm

Intermittent outage in one of our Los Angeles data centers (RESOLVED)

Posted (September 6th, 2012 at 1:15 pm PST) by danwy

We are currently experiencing a network outage in our Los Angeles area datacenters, which is causing web and MySQL downtime (email services are not impacted by this issue). Our administrators have identified the cause as network connectivity issues within our network between data centers. What this means in practical terms is that while all services are up and running, a handful of customers who have web and MySQL services in different data centers are experiencing outages as the components are unable to communicate with each other. We will make sure to update this post as soon as we have further information. We apologize for the trouble and appreciate your patience.

Update: Friday September 7th 1:00am PDT: All services and connectivity have been restored, and we will be following up with our vendor to prevent any further issues. Thank you, once again, for your patience while we worked toward a resolution.

Update: Thursday September 6th 7:43pm PDT: All services and connectivity (for shared and VPS) remain stable. However, we are still experiencing some network issues with some of the affected dedicated servers. Our admin and networking teams are working on correcting this now, so we’ll continue to leave the issue open and provide you with a final update once we’ve received confirmation that the issue is resolved. We ask that any dedicated server owners -do not- reboot or powercycle their servers until this post is resolved. We appreciate your patience while we get this worked out.

Update: Thursday September 6th 5:30pm PDT: We believe all services and connectivity have been restored. Our admin and networking teams are still monitoring the situation to make sure the network has stabilized, so we’ll leave the issue open and provide you with a final update once we’ve received confirmation that the issue is resolved. We appreciate your patience while we got this worked out.

Update: Thursday September 6th 5:07pm PDT: We have most of the affected networks rerouted at this time. There are still a handful of network paths that need updating to bring them back online. Our admin and networking teams are still working on resolving all of the issues created by this incident. We will update this post again shortly with more information.

Update: Thursday September 6th 4:35pm PDT: We have most of the affected networks rerouted at this time. There are still a handful of network paths that need updating to bring them back online. Our admin and networking teams continue to work on resolving all of the issues created by this incident. We will update this post again shortly with more information.

Update: Thursday September 6th 4:15pm PDT: Our admin team is making progress on rerouting all the affected network connections involved with this issue. We have nearly all customer services routed around the problem network. Our network operations team is still working with our upstream provider to determine a fix for the underlying issue. We’ll update this post again shortly as soon as we have more information.

Update: Thursday September 6th 3:45pm PDT: We are continuing to see improvements due to the routing changes we’re making, but sites are still down. We are still diligently working with our upstream provider to resolve this issue and will update this post again shortly with more information.

Update: Thursday September 6th 3:15pm PDT: We are seeing some improvement thanks to the workaround, but our network engineers are still working diligently to restore service for all impacted customers. We will update this post again shortly with more information.

Update: Thursday September 6th 2:50pm PDT: We apologize for the continued inconvenience. We are still in the process of applying both the workaround and final fix with our upstream provider and will provide additional information as quickly as possible.

Update: Thursday September 6th 2:30pm PDT:  We are still working with our upstream provider to resolve this outage. In the meantime, our admins are trying to route our networking around the problem in order to restore service. We will update this post again shortly with more info.

Update: Thursday September 6th 2:10pm PDT: We continue to be in contact with our upstream provider to resolve the connectivity issue. We hope to have this fixed soon and we will update this post again shortly with any new information.

Update: Thursday September 6th 1:50pm PDT: We have been in contact with our bandwidth upstream provider and have verified that the connectivity issue has been narrowed down to their infrastructure. We are in contact with their network engineers and hope to have this resolved soon. We will update this post again shortly with any new information.

Update: Thursday September 6th 1:20pm PDT: The network issues are affecting inter-datacenter connections. This is only affecting customers with web and MySQL services (no email services are impacted) that span multiple data centers which is a very limited subset of customers. Our Network Engineering team is working as quickly as possible to resolve the issues.

This entry was posted on Thursday, September 6th, 2012 at 1:15 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

50 Responses to “Intermittent outage in one of our Los Angeles data centers (RESOLVED)”

All my websites are down…. database errors…. nothing is coming up…. GAAAHHHHHH!!!!!!

WHY ARE YOU DOWN EVERY OTHER DAY?!?!

You realize that you could be the death of my business right now……?

My sites are down as well. This is happening too often.

Get me some information ASAP. What seems to be the cause?

Is this affecting mail? I am seeing delays in one of my discussion lists.

Took them 20 minutes to post this post!!!! $%#^&%$^#@!

So much for trying a new host! Why does this happen AFTER the trial period and annual payment in full?

See ya, Dreamhost. I (and, by the way, ALL of my clients) are leaving and never coming back.
Screw. You.

@Pete, just ask for pro-rated money back and LEAVE. This WILL NOT stop happening.

Are you serious? We just launched a few days ago and you’re down? WTF!

redceder I assume is in LA

Well that would explain it… I suppose I should have read this before I reset my server.. great.

Someone has to point this out every time:

If your business is ruined by Dreamhost going down for a few minutes, maybe shell out the money for a business-class hosting provider. If it’s that important, you shouldn’t be relying on a cheap, shared hosting solution to begin with.

This happens way too frequently. Please fix this as fast as possible!

I access my mail via eGroupware. In this way, the mail is down: “Connection timed out”.
When I access by webmail, it’s OK.

We’re down atm, our MySQL server is “nursehound:ahimud” MySQL Server
hope it’s back soon, has been over 20 minutes now.

Unnacceptable. we are switching off of dreamhost asap

Come on Dreamhost… This is happening way to often.

September 6th, 2012 at 1:30 pmGeorge Dickinson Says:

What is the estimated time for this to be fixed?

Oh and it would be a great idea to make sure that your twitter account (@dhstatus) is updating!

all mine (and my clients’) websites are working just fine.

It’s so depressing that Dreamhost has so much downtime. I introduced so many web design clients to Dreamhost because of its Google apps integration, I feel like crying now. I have a midnight deadline (Ireland time), automated launch invites and everything ready to send. I’ll lose a client, this month’s £450 rent. Thanks.

Another client on Dreamhost, a non-profit, had a full day of downtime, just when something important comes up. They put nothing on the blog, nothing on twitter, nothing on this site. They were so kind as to let me keep the deposit. A bit of honesty goes a long way guys :( Explain the problem! And explain how out of date the servers in the data centers are! :(

Yet another fail. It must really suck to work there and know that the entire user base wants to kill you the minute you show up for work.

This started happening at around 1pm (PST), not the reported 1:15. When is this expected to be resolved? Thanks.

Ps, all for-profit clients websites down. The non-profits had their downtime on their own accounts a week or two ago. I can’t do these late nights with all these issues, please be honest and update Twitter, and put in-depth detail here! :(

We officially launched our website TODAY by sending out 30,000 emails — and now it’s down??? That sucks!

Damn but there’s some self-important cheapskates on here.

If your site is business critical, fork out for a dedicated server, plus co-lo. Don’t rely on somewhere like Dreamhost.

From my side, nothing is business-critical. Sure it’s a pain to be down, but there’s not much I can do about it, and for my money, Dreamhost provide a better service than most.

Stop bitching. If you’re too cheap to pay for business-level (enterprise) hosting don’t fucking whine when standard hosting falls over on *rare* occasions. Christ.

Second time in less than 24 hours my site has been down. This is really stressful. Why does this happen so frequently :( I feel like the fact that I don’t want to deal with the stress of switching web-hosts is the only thing keeping me here anymore :/

i’ve requested multiple times to have my server and databases in the same datacentre, but nooooooooo, DH always assures me that there will be no more problems with having them in separate datacentres. My sites are up, but any page that requires a database connection times out. get with the program DH. put my databases next to my server!

Honestly, how can you guys afford to have this happen.

some of my incoming email has been delayed….but not all… strange… some is still missing.

Why the hell do you people put MySQL servers and web servers in different datacenters if you’re not fucking good enough to keep them on the air?

Agree with Andrew. This is stressful. The people who designed our site (and went out of business) recommended Dreamhost and transferred our site for us. If I had more technical expertise, I’d be running away to somewhere else right this second.

Refund, please. That’s absurd.

limited subset of customers???????????????? we pay the bills!!!!

Still waiting. This is very frustrating. I’ve never had an issue with my site, so I recommended DreamHost to my cousin for her non-profit. Her site is down now because of this even though mine is till up. When will this be resolved?!?

ROFL… you people crack me up.

If 30 minutes of downtime destroys your business… well you’re in the wrong business, better just pack it in and go home. This happens on the internet all the time… and real life continues… wow.

People need to relax, shit happens to computers AND servers all the time.
They will get resolved soon :-)

All my sites are down!!! Tanks for the very fast Network Engineering team.. Till xmas we are back..

@John & @Dan
Yes, shit happens, but this happened because of DH’s ill-advised decision to place servers and databases in separate datacentres. My sites are fine, except when a page needs a DB connection to render. This is 100% the fault of DH’s design decisions which they have been warned about before. I requested to have my databases and servers in the same datacentre many months back, but DH just kept assuring me that they won’t have any more problems like this. I even rewrote a lot of code to cache a lot of DB info so that short DB connection problems wouldn’t be noticed, but > 5 minutes is too long. This is 100% DH’s fault, not an accident.

Dan,

If you wish.. I could give you all my customers phones.. and you can explain it..

In the last week and a half, this has happened 3 times to us. I’m the site admin for a university library website. Our university can’t host us because we’re apparently too fancy. But it’s also a university, so, not much money in the budget for web hosting! Hence, Dreamhost. It’s not like it destroys our business, but it does make US look bad – and me by extension.

@FatPheasants, I’m not sure what your financial situation is, but for most small businesses, we have to start on a shoestring budget in order to make it. Sure, business class hosting is fine when our small business can afford it, but when you’re sacrificing a paycheck to fund your startup, every penny counts. There are a ton of budget hosting solutions that live up to their 99.99% uptime claims. Dreamhost isn’t one of them, as this is not the first time my day-job’s website has been negatively affected in the past couple of months.

I wonder how many of these “just relax, don’t be such a cheapskate” posts are from actual people, and how many are simply trolls?

ANYways, I pay good money for 2 separate VPS servers and a dedicated MYSQL server. I expect services to work.

@John – the timing of it can really suck for people. Up above, you read that an email newsletter was sent to 30,000 people announcing a new site.

Last night, my BFF released an app and her site went down due to these same reasons about the same time the app came out (her announcement emails and FB page and everything included links to the site).

Today, I sent out a newsletter and the site went down.

It’s embarrassing and unprofessional.

Do we have an ETA yet as to when the issue will be fixed?

@OnAShoestring – name the budget hosting solutions that live up to a 99.9% uptime claim! I’d like to hear about them.
If they truly exist.

@bob
So the difference in this case would be that your site would be completely down, compared to being without a database… It’s the same thing. Either way, you’d be down right now.

I will tell you this though, I’m going to be looking at setting up the cloudflare service once I’m back up.

Put our FILES next to our DATABASES. Easy as pie.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Hanjin server offline for hardware replacement (RESOLVED)
6:38 pm

Hanjin server offline for hardware replacement (RESOLVED)

Posted (September 3rd, 2012 at 6:38 pm PST) by Dan J

Our admin team have taken the hanjin.dreamhost.com server offline temporarily. They are replacing some of the hardware components used with this server due to a failure. The server will remain offline until tomorrow morning Sept 4th and backup users will not be accessible until then. No data should be lost due to this maintenance. We apologize for this unscheduled maintenance and will continue to keep you updated on the situation.

Update: Friday September 7th 1:55pm PDT: The server is now back online and functioning correctly. We did some testing and can confirm that file access and permissions are working as expected. We are keeping this status post unresolved for a little while longer to keep an eye on the service. Thank you for your patience while we make sure this service continues to work adequately.

Update: Wednesday September 6th 6:22pm PDT: At this time the hanjin server is offline while we work to replace some components. Backup users will be inaccessible until the server has been restored. Again we apologize for the issues with the backup server, we hope to have this corrected for you soon. – Dan J

Update: Wednesday September 5th 6:30am PDT: At this time the machine is back online, but not to its full capabilities. Any changes such as adding a user or changing your backup user password are not currently getting implemented. We are working on the needed repairs still necessary to get this server fully functioning again. Existing users with their current configuration should be able to work normally with their backups. Our apologies for the issues with the backup server, we hope to have this corrected for you soon. – Justin K

Update: Tuesday September 4th 1:58pm PDT: We are happy to report that the machine is back online. All services have been restored. As previously stated, your data was not impacted by this outage. Thanks for your continued patience. If you notice any problems, please Tech Support support with the details. -Brian S

Update: Tuesday September 4th 12:30pm PDT: Our System Engineering team is still working towards diagnosing the issue and bringing the machine back online. All services will remain inaccessible until that time, and we currently have no estimate how long that may take. We will post updates as information is made available. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. -Brian S

This entry was posted on Monday, September 3rd, 2012 at 6:38 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Border/Core Router Software Upgrades (Los Angeles Data Center) Thursday August 23rd 8pm PDT (4 hrs) (RESOLVED)
5:40 pm

Border/Core Router Software Upgrades (Los Angeles Data Center) Thursday August 23rd 8pm PDT (4 hrs) (RESOLVED)

Posted (August 22nd, 2012 at 5:40 pm PST) by andrea

After successfully completing the Border/Core Router upgrade at our Orange County and LAX Data Centers, we’re moving on to the Los Angeles Data Center. The maintenance window will start at 8pm and last for 4 hours on Thursday, August 23rd. There is no expected service interruption during this upgrade as there are redundant routing engines in place >on the routers that will help us avoid service interruption. This process functions by having the non-active engine upgraded first, then brought online so the others can be upgraded while inactive. We will keep you updated on the progress as we begin this maintenance window.

If you have any service issues before the maintenance window, please contact support for assistance through the DreamHost panel.

Update: Monday August 23rd 9:15pm PDT:  Maintenance has completed with no downtime! Tomorrow, we’ll be performing more of the same to get our network in tip top shape!

Update: Monday August 23rd 9:15pm PDT: We’re still seeing everything running along smoothly, no downtime as of yet! Hardware is still being upgraded and taken online.

Update: Monday August 23rd 8:50pm PDT:  We’re now starting on the upgrade of core router hardware. We’ll keep things updated as we progress.

Update: Monday August 23rd 8:35pm PDT:  Ok, we’re all done with the upgrade of our border network hardware. Everything went just fine. If you’re seeing any problems, please contact support for assistance through the DreamHost panel.

Update: Monday August 23rd 8:05pm PDT: The maintenance is about to begin, our network engineering team and several of our system administrators and datacenter team members are at the ready. We will be updating this post with the progress of the maintenance.

This entry was posted on Wednesday, August 22nd, 2012 at 5:40 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network maintenance – component replacement and software upgrade (Los Angeles Data Center) Friday, August 24th 8pm PDT (4hrs) (RESOLVED)
6:16 pm

Network maintenance – component replacement and software upgrade (Los Angeles Data Center) Friday, August 24th 8pm PDT (4hrs) (RESOLVED)

Posted (August 21st, 2012 at 6:16 pm PST) by andrea

In accordance with our current network maintenance philosophy, we have been keeping a close eye on our newly installed systems in all our datacenters. In our smallest one, we discovered a component which doesn’t appear to function as expected. While it’s not causing any service issues, we decided to get it checked out, just in case. We are working with the vendor to ensure seamless replacement and software upgrade of that component. We set our maintenance window to 4 hrs, but we anticipate no more than 15 minutes of actual downtime. We will be monitoring the comments during the maintenance, and will keep you updated about the progress.

If you have any service issues before the maintenance window, please contact support for assistance through the DreamHost panel.

Update: Friday August 24th 8:17pm PDT: Maintenance has completed with no downtime, and in record time! Everything is showing green on our end, and that concludes tonight’s updates to our network topology.

Update: Friday August 24th 8:05pm PDT: The maintenance is about to begin, our network engineering team and several of our system administrators and datacenter team members are at the ready. We will be updating this post with the progress of the maintenance.

This entry was posted on Tuesday, August 21st, 2012 at 6:16 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Border/Core Router Software Upgrades (Orange County Data Center) Monday August 20th 8pm-Midnight (RESOLVED)
3:23 pm

Border/Core Router Software Upgrades (Orange County Data Center) Monday August 20th 8pm-Midnight (RESOLVED)

Posted (August 17th, 2012 at 3:23 pm PST) by Ralph

This coming Monday, August 20th, we will be upgrading the software on our Border and Core routers at our Orange County Data Center. The maintenance window will start at 8pm and last for 4 hours. There is no expected service interruption during this upgrade as there are redundant routing engines in place on the routers that will help us avoid service interruption. This process functions by having the non-active engine upgraded first, then brought online so the others can be upgraded while inactive. We will keep you updated on the progress as we begin this maintenance window.

Update: Monday August 20th 11:48pm PDT:  We have successfully completed this hardware upgrade, without service interruptions. We are continuing the network maintenance series in our other data centers, so check back here for upcoming status posts. Thank you for your patience while we’re continuing to improve our services!

Update: Monday August 20th 11:14pm PDT:  The maintenance is progressing as expected. We will update this post as soon as the status changes, or within an hour, whichever is less. We appreciate your patience.

Update: Monday August 20th 9:40pm PDT:  We have finished upgrading the borders with success, and started the core router upgrades. We are working with the new router vendor to ensure a smooth transition. As expected, we experienced no service interruptions.

We will update this post again within an hour or sooner if a status change necessitates it.

Update: Monday August 20th 8:40pm PDT:  So far so good: we started the maintenance and all the periodic checks are showing no issues. We will update this post again within an hour or sooner if a status change necessitates it. If you have any service issues in the meantime, please contact support for assistance through the DreamHost panel.

Update: Monday August 20th 8:00pm PDT: The maintenance is about to begin, our network engineering team and several of our system administrators and datacenter team members are at the ready. We will be updating this post with the progress of the maintenance.

This entry was posted on Friday, August 17th, 2012 at 3:23 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Border/Core Router Software Upgrades (Orange County Data Center) Monday August 20th 8pm-Midnight (RESOLVED)”

I have one problem , because my server 4wsites.net dont function more.

I need Help !!

I have been experiencing very slow load times after this upgrade, for both the Admin Page and the actual site. When I load other sites like Facebook, Yahoo, Dreamhost.com….no problem

Shared server sites much slower after this upgrade. 10 second page load times. Maybe just a coincidence? Support taking hours to reply to a support ticket on the matter.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – Core Upgrade Part 3 (Los Angeles Data Center) – Friday August 17th 8pm PDT (4 hrs) (RESOLVED)
5:31 pm

Network Improvement Series Maintenance – Core Upgrade Part 3 (Los Angeles Data Center) – Friday August 17th 8pm PDT (4 hrs) (RESOLVED)

Posted (August 15th, 2012 at 5:31 pm PST) by andrea

Our Network Improvement Series Maintenance will continue on Friday, August 17th, with an adjustment to our core router configuration. Due to topology changes, we are removing a piece of hardware that is no longer needed in the routing process. We completed the required testing and staging but will be ready to roll back the changes if necessary.

No customer-facing service is connected to that hardware, we just wanted to keep you updated about our network maintenance progress.

We will update this status post throughout the maintenance and will be monitoring comments. If you have any service issues before the scheduled time of the maintenance, please contact support for assistance through the DreamHost panel.

Friday August 17th 8:25pm PDT:  Quick and easy! As we expected, no customer facing downtime occurred, and we’re all done!

Friday August 17th 8:05pm PDT:  We have begun this maintenance window and so far, haven’t seen any customer facing issues. We don’t anticipate any at all, but will continue to monitor the network for problems.

This entry was posted on Wednesday, August 15th, 2012 at 5:31 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Upgrading web server email relay machine – August 20th, 12pm PDT (4hr) (RESOLVED)
4:51 pm

Upgrading web server email relay machine – August 20th, 12pm PDT (4hr) (RESOLVED)

Posted (August 15th, 2012 at 4:51 pm PST) by andrea

Monday, August 20th, starting at 12pm PDT, we will be upgrading the server that relays email generated on web servers. This includes, but not limited to, emails sent through phpmail and mail forms used on web sites.

We’re setting a four-hour maintenance window, but we anticipate only a very short delay in mail delivery while we move the services to a better server. No mail will be lost, only slightly delayed. This maintenance will not affect incoming mail or mail sent through email clients (including webmail) or other services, such as web or MySQL.

We will update this status post throughout the maintenance and will be monitoring comments. If you have any service issues before the scheduled time of the maintenance, please contact support for assistance through the DreamHost panel.

Update: Monday August 20th 8:48pm PDT: The maintenance is now complete. Thanks everyone for your patience.

Update: Monday August 20th 6:24pm PDT: The maintenance is nearly complete. We are just past 85% complete on the final transition to the new machine. This is the last step in the process so it shouldn’t be much longer.

Update: Monday August 20th 5:23pm PDT: The maintenance is nearly complete. We are roughly 70% complete on the final transition to the new machine. This is the last step in the process so it shouldn’t be much longer.

Update: Monday August 20th 4:31pm PDT: The maintenance is running a little longer than expected, but we can already tell that the upgraded service is processing mail faster than the old one. This means even if you’re experiencing slight php mail delays, it shouldn’t be much longer before they arrive to their destination.

Update: Monday August 20th 3:38pm PDT: The maintenance is almost complete, and we didn’t run into any unexpected issues. Sorry about any delivery delays. We’ll update this post again as soon as the machine upgrade is complete. Thanks for your continued patience.

Update: Monday August 20th 1:38pm PDT: The maintenance is under way, and so far so good. You may be noticing delay in delivery, but no mail will be lost. Please consider the maintenance window, and let us know if you need any help by contacting support for assistance through the DreamHost panel.

Update: Thursday August 16th 3:00pm PDT: The setup on this was delayed. The new server has been created now and we are testing it before replacing the current email relay. We have rescheduled the switchover to Monday afternoon between 12pm and 4pm PDT.

This entry was posted on Wednesday, August 15th, 2012 at 4:51 pm and is filed under System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connection issues on some of our mail servers
2:08 pm

Connection issues on some of our mail servers

Posted (August 6th, 2012 at 2:08 pm PST) by danwy

We are currently experiencing some issues with one of our mail load balancers. This is a machine that routes traffic (emails) through our mail clusters and as a result this is causing some disruption of mail and webmail services. Details are still developing at the moment and our admin team is already hard at work on the problem. If you are having issues sending or receiving mail at this time, it is likely related to this matter so please give us some time to get it resolved before continued testing. We will update this status post as soon as we have any information and thank you for your patience.

Update: Thursday August 6th 2:30pm PDT: The load balancer being investigated is currently under very heavy load which is leading to service disruption. Our admin team is in the process of identifying and correcting the source of the abnormal workload being put on this machine. We’ll update this post again as more information becomes available.

Update: Thursday August 6th 3:00pm PDT: We have resolved this issue and mail appears to be functioning normally again. Our admin team is still correcting a configuration issue which prevented our redundant measures from properly executing but the service disruption itself is resolved. We apologize for any inconvenience this caused you and thank you for your patience.

This entry was posted on Monday, August 6th, 2012 at 2:08 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

19 Responses to “Connection issues on some of our mail servers”

first!

again, as always

Thanks for the report.

First! (And I might have been the first to tweet them!)

Damn it… fourth :(

Looks like it’s time to migrate mail off of DH servers as well.

I still got failed result…:(

Come on this is not working out to well lately and it needs to be addressed soon or we will be asking for refund and going somewhere else for hosting our email and web stuff..

August 6th, 2012 at 2:12 pmAndy Ringsmuth Says:

I think I was first to report it! Via chat at about 1:38 p.m. PDT (the same time used in these forums) :)

Either I am just the luckiest person in Dreamhost land, or their “couple of mail clusters” is really the whole damn network. I am down every freakin time. Why I stay is beyond me

August 6th, 2012 at 2:15 pmWhite House IT Says:

Thank you, Dreamhost for completing my Monday. You……complete me…

It seemed like homiemail-sub4 don’t play that. But now he seems to be okay.

come on dreamhost. This is nightmare.

Now it’s up and running for me. Thanks fro your prompt address. :)

Well it’s back up for me now, but I’m showing that I received no messages at all from 13:14-14:10. That’s okay, I’m sure there was nothing important sent during that time…

Its working for me right now…

Dreamhost performance for the past few months is sub-par at best. Our clients seem to be hit with every rock in the dreamhost road. (Multiple client domains down today again!)

August 6th, 2012 at 3:06 pmAndy Ringsmuth Says:

Cut ‘em a little slack. Going from the notification of the problem to identifying it and fixing it in about half an hour is pretty good. Any of you ever tried self-hosting e-mail? It’s a nightmare. I used to self-host several domains and I hated it. Once I migrated to DH a while back, it’s been wonderful. Yes, there is the occasional hiccup, but that WILL HAPPEN no matter where you have your e-mail. It doesn’t matter how much you pay, there WILL BE downtime occasionally.

If Google, Hotmail, Amazon, etc., can go down (and they do from time to time) then it can happen to anyone.

Went out for me at 1:30pm PT, but couldn’t post because there was no issue to post to, then. I just went to Mail2web for a while, which was working fine, until my webmail server came back. Still, would prefer not to have to do that — the format is slightly different and confuses the consistent look of my messages.

Also, I tried to find the control panel feature where I can test my systems, but can’t find it anymore — is that still there somewhere?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – Core Upgrade Part 2 (Los Angeles Data Center) – Thursday August 2nd 8pm PDT (6 hours) (RESOLVED)
4:46 pm

Network Improvement Series Maintenance – Core Upgrade Part 2 (Los Angeles Data Center) – Thursday August 2nd 8pm PDT (6 hours) (RESOLVED)

Posted (July 31st, 2012 at 4:46 pm PST) by Ralph

Our Network Improvement Series Maintenance will continue on Thursday, August 2nd with the remainder of our upgrades. We have roughly five more racks left to switch over. The upgrade on Friday was successful. These five racks were not completed with the Friday maintenance as we did not want to push the maintenance window much further than we had already posted. We were able to minimize any downtime experienced to less than two minutes each rack. Our expectations are the same as last Friday, which are listed below.

The maintenance on Thursday will switch over specific subsets of our data center. This means that the move will happen in small chunks rather than the full data center at one time to minimize the impact on live services. In our previous upgrade we were able to limit downtime to these subsets to under two minutes each. If service interruption for a specific subset exceeds five minutes we will roll back the changes.

When the maintenance starts we will be monitoring comments and providing you with periodic updates as we progress through our process.

Update Thursday August 3rd 12:15am PDT: We have successfully completed the network maintenance! We had a slight bump in the process, but we were able to work out the issues relatively quickly and we are confident that the problem will not return. This is a huge step for us as we wind down our Network Maintenance Series and we thank you for your patience throughout. This maintenance is now considered resolved, so please Contact Support to report any issues you encounter with your services, as they are unrelated and should be investigated.

Update Thursday August 2nd 11:15pm PDT: We are still making progress and seeing only slight interruptions on racks we have completed.  These slight interruptions are the expected result and what we have been seeing in the majority of our rack moves.  The interruptions last less than 30 seconds.  We continue making good progress right now and are feeling confident in the completion of these upgrades.

Update Thursday August 2nd 10:45pm PDT: We are making solid progress as we continue from rack to rack.  We are seeing very limited service interruptions in our past few successful rack moves.  We will continue to update with our progress, but everything is coasting right now.

Update Thursday August 2nd 10:30pm PDT: We are continuing with our maintenance and are progressing onto the next row.  Despite the unexpected disruption, our confidence is high that the remaining racks pose no similar risk.  After we got service restored we were able to successfully move the next row on our list.  We are now progressing through another row of racks at this time.

Update Thursday August 2nd 10:25pm PDT: Our connection to the datacenter appears to have been fully restored. Our network engineers are still monitoring the situation to make sure we are in the clear.

Update Thursday August 2nd 10:03pm PDT: Our connection to the datacenter appears to have been fully restored. Our network engineers are still monitoring the situation to make sure we are in the clear.

Update Thursday August 2nd 9:41pm PDT: We have partially restored the connectivity to this datacenter. We’re however still exhibiting some unforeseen issues related to the outage. We’ve mobilized everyone and are actively investigating. Network engineers are already onhand for the maintenance. More details to follow.

Update Thursday August 2nd 9:14pm PDT: We are currently experiencing a network outage in our Downtown datacenter, which is causing web, MySQL, and email downtime as well as access to the Dreamhost panel and Dreamhost.com. Our administrators are working on identifying the cause. We will make sure to update this post as soon as we have further information. We apologize for the trouble and appreciate your patience.

Update Thursday August 2nd 8:42pm PDT: The second switch reboot is complete.  We are now moving on to the next switch.

Update Thursday August 2nd 8:35pm PDT: The first switch is complete, there was a very brief mysql service connection outage while it was rebooting.  We are now moving on to the next switch.

Update Thursday August 2nd 8:11pm PDT: We are now starting with the first row of machines. Machines on this row will experience a short service interruption.

Update Thursday August 2nd 8:08pm PDT: Physical cabling is underway as we prepare to start on the first set of machines. To this point there has been no service downtime.

Update Thursday August 2nd 7:55pm PDT: Our team is assembled and getting ready to start on this maintenance. As we progress through the maintenance we will provide periodic updates and assist you both through your comments and through our support system.

This entry was posted on Tuesday, July 31st, 2012 at 4:46 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

172 Responses to “Network Improvement Series Maintenance – Core Upgrade Part 2 (Los Angeles Data Center) – Thursday August 2nd 8pm PDT (6 hours) (RESOLVED)”

everything down now

Does this mean my website can not be accessed in 6 hours? it is terrible.

All down here too.

Cut off here too…

what the fuck? another downtime? YOU SUCK

down for me too

Me too. Does this happen with other hosting companies?

Everything is down for me too.

August 2nd, 2012 at 9:00 pmGerman Hernandez Says:

Guys, something is seriously wrong. My sites are down and are at the Irvine datacenter. I was planning on submitting a new ticket for there is no access to the dreamhost panel.

everything is down now.

Down here too since 8:53

My sites are all down. Glad I moved the important ones to InMotion

mysql error was: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 113

site down for over an hour now

This is my last week on Dreamhost. Can’t wait. Fucking idiots couldn’t serve a tennis ball.

You all do know… ALL HOSTING COMPANIES HAVE TO DO THIS. It’s what happens with computers. I’ve had multiple hosting companies and they’ve all been like this. It’s not that difficult to understand. If you’re having lots of downtime, report it. I’ve been using DreamHost for 6 years and have rarely had downtime. If you’re afraid of downtime, check out CloudFlare.

Yup, down right now.
But i mean, they are updating things, it happens with all hosting companies at some point in time.
But i do have to say, dreamhost has the best support team out of all the others hosts iv used.

YOU GUYS ARE FUCKING KILLING ME. I’M TRYING TO RUN SEVERAL POLITICAL INFORMATIONAL WEB SITES AND YOU KEEP GOING DOWN. PRIMARY ELECTION IS ON TUESDAY YOU IDIOTS. WHY IN THE H*ll DO YOU KEEP DUMPING YOUR SERVERS. JESUS F*CKING CHRIST. I GET MOST HITS DURING THE GOD DAMN EVENING YOU CLOWNS!

Down here – again. This is really silly – hope I can get a refund as I just re-upped for two years

Panel down! Panel down! I wonder if this is related to the upgrade and just a temporary outage or has something gone horribly wrong.

Went down right as I was working on something over SFTP. Just great.

Yup.
Website: Down.
Email: Down.
Control Panel: Down.

I live in San Francisco and my email is down. From your update I thought it was just in LA…. what is going on?

What’s Up? You’re not. You fucking twits.

Sure, all hosting companies need to do updates, but why isn’t downtime scheduled for the middle of the night?!

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

My site down too

WEEPING JESUS ON THE CROSS WILL YOU GET ONLINE YOU GUYS!

@ThiAvr

Because it’s different times in all parts of the US… you do understand basic time zone concepts, right?

hahaha third time (switch reboot) wasn’t the charm?

Tell us how you really feel, Adam.

DO IT ON CHRISTMAS. DO IT AT MIDNIGHT PDT. DO IT NOT DURING MAJOR EVENTS – PRIMARY ELECTION WEEK.

ANGRY ADAM NEEDS BACON. A POUND OF BACON. I’M GOING TO COOK UP A POUND OF BACON AND THE TAKE A NAP. I’M KAPUT.

August 2nd, 2012 at 9:08 pmGerman Hernandez Says:

even http://www.dreamhost.com is down!

Webmail is down, site is down, email is down, ftp is down, the panel is down, even dreamhost.com is down!

Did someone forgot to pay the eletric bill???

Well at least my favorite website is up. Click on my username. I wonder who they host with?

Surely the external SSH interfaces to the subnets being migrated could have been maintained. I’ve been unable to log in via SSH for the past hour.

-patpat- there, there…

Down here too! Not sure how long though.

Thanks for all the hard work guys. I appreciate you letting us know about the maintenance. Ignore the trolls with unrealistic expectations, clearly they haven’t worked in a Datacenter. I’ve hosted with you guys for six/seven years now and have always experienced excellent service. Keep up the good work.

EVERYTHING IS DOWN!!! PANEL, SITES, DATABASES, DH Home Page, EVERYTHING!

So how come this time you weren’t able to minimize impact to under 2 minutes each?

August 2nd, 2012 at 9:10 pmDonald Rasmus Says:

same here, no web sites, no web panel, no love. :( sigh.

My sites aren’t down, but my mail and panel are.

Everything is currently down for me.

Joe, can you show your data? Because I used a web site monitoring service that reported downtime that occurred for 5-minutes or longer that says otherwise, over a the last 3-years. According to my data, at least one 5-minute, or longer, outage occurred every 3-days.

In contrast, I have run the same web site monitor on my Linode.com-hosted sites over the past year and there was a single outage that lasted for 6-minutes over the course of the entire year.

The conclusion: Dreamhost isn’t the best place to host important websites.

Thanks for all the hard work guys. I appreciate you letting us know about the maintenance. Ignore the trolls with unrealistic expectations, clearly they haven’t worked in a Datacenter.

My site has been down three times today. Go fuck yourself.

I’ve been with Dreamhost on one of their Unlimited plans since 2006 and in the past two years there have been quite a lot of outages. I know it’s a relatively cheap service but I do expect better than an almost monthly or bi-monthly service interruptions.

I’ve been unfortunate that I renewed my hosting for the next two years no more than a week ago; I should have renewed for one year, as I’m starting to get bitterly disappointed with the consistent outages. I’ll give them the next two years to fix it and if they can’t maintain a higher level of uptime, I’ll look around to see if anyone has load-balanced fail-over servers.

It would also be beneficial if they considered some data-centres in Asia to load balance visitors (my latency is pretty bad, but I chalk that up to being in Australia).

Everything is down here. :/

Folks, websites are down and no panel access to contact you through the official channels. Where is the message about this?

all my site down now! really sucks! time to move!!

My site has been down three times today. Go fuck yourself.

+1 ;)

@Jake: They don’t always notice things like that right away. Give it a bit.

Still all down

If you are having that much down time why would you not contact your hosting company? I had alot of downtime with my email a few weeks ago and they kindly reset everything for me quickly. It’s a shame that the good companies have to put up with such rude customers.

My sites are all down.

Yep, down. sigh.

I’m seeing “all down” messages and tweets for 20 minutes now. Wonder what DreamHost considers an acceptable SLA.

@Pat Farling

I’m seeing “all down” messages and tweets for 20 minutes now. Wonder what DreamHost considers an acceptable SLA.

All my sites are down and I’m unable to load the control panel as well. Please update the status of this notice to let everyone know what is going on. Last time the outage was minutes – this has been going on for 20 minutes or more now, and no updates. Please let us know what the status is.

Thanks

AEST 2:13pm Friday 3rd Aug

Hey Guys – looks like the Dreamhost is completely offline
I cant get to any of my client websites
And I can’t even log into Dreamhost to raise a support ticket

this is CRITICAL – as all my clients will be without email

Please provided an update on dreamhost status

I’m themeforest author, i feel very disappoint with dreamhost services! all my preview down!

2 minutes? Mine have been down for at least a half hour so far. By the way, I used the official channel a month or so ago about getting reimbursed for this down time and never got any response. You say you are monitoring comments, please comment on this.

If you are having that much down time why would you not contact your hosting company? I had alot of downtime with my email a few weeks ago and they kindly reset everything for me quickly. It’s a shame that the good companies have to put up with such rude customers.

Site down. Email down. Support down. Twitter unresponsive.

I stand by my previous comment: Go Fuck Yourself.

By the way, you all do see when this was posted right? You had time to notify your customers and viewers. It’s not that hard to subscribe to an RSS feed. That’s what I do and it works out pretty great.

Email is working now.

It’s back up. Relax, guys.

Just spoke to tech support @ Dreamhost, even they cant do basic checks on the servers. This outage has rocked even them. They cant offer any insights, or even confirm if outages are connected to this issue.

So what ever is going on its big, its rattled the support guys and disabled them form being able to provide ANY confirmation of the outages or do any sort of checks.

I have to say in 7 years hosting, 2012 has been Dreamhosts worst ever for outages, and mostly these outages followed ‘PLANNED MAINTENANCE’.

My Panel is back up now but the sites are still down…

probably like when you take something apart and put it back together, there’s always one thing left over: “So, where do you think *this* wire goes?”

My Panel is back up now but the sites are still down… Probably hasn’t propagated everywhere on the Internet yet peeps.

When I do a trace route, I get to newdream.net and then everything times out. Yea, seems like something wrong internally. We’ll see what they blame it on this time.

Still down… I see you got dreamhost.com up though.

My web sites and email server are down, but dreamhost.net is up.

How am I supposed to run my business with hosting service like this?

So I find it rather shady that when this first started even dreamhost.com was down; however, now it’s magically running again. Perhaps you guys should be spending your time fixing the network issue that has taken all your customers offline rather than making sure your own website is up and running. :@

And seriously guys, who schedules “Network Improvement Series Maintenance” at 8PM PDT? Isn’t this a middle-of-the-night-while-most-of-our-customers-are-sleeping thing so that when shit hits the fan, as it obviously did tonight, far fewer people are affected — or even know that there was a problem?

All sites are still down. :((. All three of them.

God save the Queen!

(checking if email really is down…)

all sites are down..god!

Forgot to mention that my webmail is up although my sites and mail are down. Interesting.

@Jeremy- the internet never sleeps. No matter what time it occurs, a good number of people are gonna be affected. It’s always a peak hour somewhere.

Thanks for keeping us updated, ignore the idiots.

@Martin welp, if you have a business worthwhile, you’d get real hosting for it. If you can’t afford it, then this is honestly speaking, not all that bad. Annoying, yes, but not the worst thing in the world to happen.

Holy fuck, grown adults acting like children. Calm down people, really.

Everyone who complains about the down time is a moron… I was going to suggest some stuff but whats the point. Based on the dumb ass comments you wouldn’t understand what I was going to say anyway……..

Funny thing, I just now got email from HDI asking how well Dreamhost Technical Support handled my report of the outage a couple of days ago.

Oh the irony…

I might make the suggestion however, that you have the foresight to have reduncancy in another data center to avoid issues like this.

@Josh Agreed!

@DJ ALLYN

LOL i know youre reading, i have just about finsihed my move from all my VPS and will kill the switch on this crap company.

Next after that will be the shared accounts.

Good riddance to scumbags like you (DJ ALLYN) and all others who stick up for this pathetic excuse for a hosting provider who know nothing about reliable hosting and companies like DH who are too busy whistling out of their but on facebook and twitter and buying think geek toys for ther desktops to actually do anything productive about their hardware and network!!!!!!!!!!

But believe me, I’ve seen far worse. Just try that place with “fanatical support” sometime.

my site is down it’s on spiderman i need it up by tomorrow morning or the world will end. please.

@mike john josh

dh is crap

“it’s on spiderman i need it up by tomorrow morning or the world will end.”
Sounds like an intro to a bad, slashy fanfic.

“Sounds like an intro to a bad, slashy fanfic.”

So true. :D

@jimbo
Better than others out there for the price. Good Fast Cheap, pick two.

@jimbo

Everywhere that has cheap hosting is crap. Hence the cheapness of it.

yep, my stuff is down too. Looking forward to it coming back :-)

Sh*t happens, calm down people, haha! I’ve never had a host that had zero down time, I don’t think it’s possible.

I’m convinced that the only metric Dreamhost uses to determine if a service is down is the activity of comments on this site. I wonder if XKCD has a cartoon about this yet.

Simon: “Hey, we’re getting a few comments regarding EVERYTHING IS DOWN. Should we look into this?”

Josh: “Naw, we’ve got a good ping-pong game going, let’s wait until after the game to check”

Simon: “Yeah, maybe the problem will resolve itself”

[Line graph showing larger spike in comment number]

Simon: “Josh, okay, now the comments are really pouring in, I think it’s time to have a time-out at ping-pong and look into this”

Josh: “Oh, something MUST be wrong! QUICK: Alert Ralph to create a post on dreamhoststatus.com and I’ll drive out to the data center to see what’s up”

Simon: “It’s rush hour right now. They’re going to be upset. I should address this during our next cute newsletter”

Josh: “Yeah, that’ll win ‘em over. It’s that Australian accent that I think they love so much. People will complain, but only a small percentage ever leave. It’ll be okay.”

hi guys

Again down 2 times this week , what is happening ? next time please advice the scheduled window because I need to advice my users too. And next time if you can , please move the schedule for midnight PDT. Thank you

What ever happened to the idea of mirroring accross multiple servers?

All my WordPress sites are giving me Database errors. Anyone else have this problem?

Allen, what’s that?

–Dreamhost Simon

ouch, right in the middle of finishing a project. should i stay up all night waiting for server to go back online (12:30AM EST now) or wake up early to finish the creative process? guess i’m part of the small chunks of database upgrade… go dreamhost go!

SLEEP IS FOR THE WEAK, JEFFFREE

@mike @josh

you guys must be imbeciles – i pay for several services with with extra ram vps etc… and all of it has been s##t for months now. Up dwon all around.

These guys dont know whether they’re coming or going and mike and josh probably work for em. go away and let us be pissed , if you loe em so much then go and do your work.

Oh thats right, you cant the host is crapped it again

what time is it

@Allen

You need to mirror across multiple sites, not just servers. Which usually means you are paying more for hosting.

Thanks for letting us know Dream Host! I love you guys, keep up all of the hard work and ignore these ignorant customers who don’t get data hosting.

JeffFree, it’s never okay to be self-important.

@Homer

love it!

if this keeps up I am going to have to do laundry which is not my personal choice

@jimbo

No, I just host my important stuff myself off my own connections and metro cluster across multiply locations so I am not affected by downtime at one location.

@jvc- what about the dishes??

i have a dishwasher

Calm down everyone and show some maturity… yeah it sucks, but outages happen, It doesn’t matter where you get your hosting from.

and the trash?

6hrs. My god. I pay too much for hosting here.

So everyone is down, not just me? That’s good at least…

trash is in the laundry room – not currently that big of a deal *looks at litter box*

I’m an understanding and patient person. I’ve been around. I know from painful experience that migrating systems is difficult and subject to unexpected problems. But Dreamhost: get your f***ing act together, already. This has long since become some seriously amateur-hour BS. For real. Hey, here’s tip: did you check to see if it’s plugged in? Sheesh.

@josh

HA HA LOLOLOLO HA HA

so brainy of you, i wish i was so clever like you.

I just have accounts with companies like Knownhost which have never had downtime for the 18months i have had several accounts with them.

Dreamhost was suggested by a friend who wanted some credits for it and i threw up some smaller sites on shared and vps but for the last 5yrs , each year has been worse. so you just keep going on boy and i’ll have to %^$^ %$$&$ KHBHBT^^ $$###!@#

@Schlank so, you can go pay too much somewhere else…

DH: Update Thursday August 2nd 9:41pm PDT: We have partially restored the connectivity to this datacenter. We’re however still exhibiting some unforeseen issues related to the outage.

Yeah, the problem is the outages we have seen in 2012, have been from planned maintenance. So here’s a tip, next planned maintenance, expect an outage!

well the litter box is clean – still down?

all sites are still down!!!

my site are down!
So weak is the dreamhost!

AHHHH!! WE WERE IN THE MIDDLE OF A LIVE DJ STREAM!!! WHYYYY!!!!?!?!?!??!?

oh chrome update – guess I can do that

so how about that laundry, jvc?

well – chrome update is done – laundry can wait – just one more day in the week right?

doh! was updating the blog for the webcomic when my site crashed! …looks like it’s just Dreamhost wetting the bed (as usual). :]

BECAUSE THE NETWORKING NINJAS HAVE BEEN CONSPIRING AGAINST YOU, SNUGGS. OBVIOUSLY.

@Ki – hah!

@Homer – d’oh!

Request timeout for icmp_seq 1042

“Calm down everyone and show some maturity… yeah it sucks, but outages happen, It doesn’t matter where you get your hosting from.”

That is so false man. Rackspace and Amazon do not have regular downtimes of 6 hours. They do not have their CPanel go down.

Access to lansky and all associated sites still down…

So who hosts dreamhoststatus.com and where do I send my check? :)

regarding laundry postponement… at least there is always febreze.

Ummmm…Are you out of your minds? How about midnight on a Saturday for this shit? Also, my site is down. Who are the dumbasses defending Dreamhost? Employees?

64 bytes from 69.163.191.132: icmp_seq=1153 ttl=52 time=73.661 ms
!

yes, down as well, but these things with DH are usually brief.

well – that was fun – you guys should have started later – bye!

HOLY SHIT, BACK UP!!!! YAY!

barker.dreamhost.com is up. Along with nerdwatch.com.

great – just launched my new portfolio site and now it cannot be accessed by the recruiters it was sent it to. dreamhost blows!!!!! we should all be credited significantly. unacceptable.

Aaaaand we’re back. 45 minutes of downtime for me, so there’s that 99.9% uptime.

I host it off a secret server I slipped in when I helped moved 150 of their servers from softaware/digital island colo a long long time ago. It works because they don’t know about it. All there problems also are caused from that move that happened decades ago.

If I had to guess, I’d say it’s a problem with the nameservers.

I’m back, thanks!!

My website just came back online after nearly an hour down.

I am glad its up (we will see if it lasts) however, downtime that approached an hour is extremely frustrating.

Chip Strong makes a good point. Why does DH mess with all these things on a weekday? My site stats say my lowest number of visitors are on Saturdays. Also, I update my website on weekdays.

Back too, thank you :)

Back up. Nice. Wasn’t long. Good work, chaps!

@Nathan
Well there’s some stuff that has to be taken care of to prevent extended downtime later, and even though you say your lowest traffic is on Saturday, and yes, they could do this in the middle of the night… that’s not the same situation for everyone who is hosted. So the obvious choice is to try and please everyone and take forever to get things done, or just get things done when they need doing.

I check email and found nothing in inbox about this. It sound sad. I love Dreamhost but why Dreamhost don’t send any email about this to their account users? Ok, at least, my site is working now.

Down still Now! “guppy:tynan” MySQL Server

Thank you for the patience guys. We have everything back in order and are continuing with the maintenance window.

OMG HE SAID WE’RE PATIENT, YOU GUYS!!! \o/

guys?

“guppy:ethridge” MySQL Server . Down still!

^ Sounds legit

that was @Homer

Guppy is definitely having issues @fisher. We are working on that machine and you should have an email from us as well as a message on your panel. Guppy’s issues are not related to this maintenance.

WE LOVE YOU RALPH! – At least the cool kids do and that’s all the matters! Right?

As the club president of the cool kids, rock on!

this time to move guys, we cannot run business with this conditions! really dreamhost sucks!!!

@ralph thanks. I’m still awaiting for the Guppy online.

All my sites are down, server “penguin”

@fisher it is being actively worked on. @Iann I have my staff checking on that. example domain?

@Iann – we checked the machine penguin and it was not having issues. Please contact support for assistance as comments are going to close on this post shortly. We are about to resolve this maintenance.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Outage in LAX Datacenter (RESOLVED)
5:36 pm

Network Outage in LAX Datacenter (RESOLVED)

Posted (July 30th, 2012 at 5:36 pm PST) by Seohee

We are currently experiencing a network outage in our LAX datacenter, which is causing web, MySQL, and email downtime. Our administrators are working on identifying the cause. We will make sure to update this post as soon as we have further information. We apologize for the trouble and appreciate your patience.

Update July 31th, 2012, 1:46pm PDT:  Thank you for your patience as we investigated the cause of the outage that affected your services last night. Our admins have completed their analysis of the incident and have determined that it was due to a hardware failure in one of our routing devices. This resulted in a loss of connectivity between our LAX datacenter and our bandwidth providers. We rebooted the machine that was having issues and that restored connectivity to the affected networks. Total downtime was just over an hour.

We have already contacted our hardware provider and we will be getting a brand new router to replace this one with. As soon as we have the new hardware, we will be swapping out the router that failed. If we expect any downtime as a result of the swap, we will post a maintenance window on our status page in the near future. Again, we apologize for any inconvenience this outage caused for you. If there’s anything else we can do for you, please don’t hesitate to contact support and let us know.

Update July 30th, 2012, 6:12pm PDT: The issue was due to loss in physical connectivity to our LAX datacenter. We are confident that the connectivity is back, and the issue is resolved. We will provide you with a final update once we have the specifics. Thanks again for your patience. ~Seohee

Update July 30th, 2012, 5:49pm PDT: We are still trying to determine the cause of the network outage, and will continue to keep you updated as we receive more information. ~Seohee

 

This entry was posted on Monday, July 30th, 2012 at 5:36 pm and is filed under Email Issue, General Outages, Major Outage, Multiple Server Issue, Mysql Outages, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

90 Responses to “Network Outage in LAX Datacenter (RESOLVED)”

Hello,

It seems that actually my website as well as SSH connection are not available, which would indicate that the outage affects more than MySQL and email. Do you have any information about it?

Yep, all services down for my site. If I leave work early can I blame it on you guys?

Been flakey all day today.

C’mon…

Can we get a list of servers affected? Just want to make sure my kraz issue is the same one.

all my sites doooooooooown help

Here we go again. My site is down.

I’m down now too. Argh.

Oh well… all my sites and VPS are down again

All of my sites are down.

8:38pm est.

This is becoming problematic. How can I run a business with this many service interruptions?? Very disappointing.

claudus is down

All my sites are down, bit annoying but hopefully it won’t be too long before it’s back.

My SQL server is ‘daggernose’ and my web server is ‘tashkent’.

Bolkens seems to be down for one hour at least.

Sabretooth is down.

Well at least I know why my site is down…

What’s the ETA to resolution?

zaniah as well.

when will be back? totally down

quote DH- “We apologize for the trouble and appreciate your patience”

LOLOLOLOLOLOLOL i am seriously pulling my hair out screaming and laughing like a mad scientist!!!

I have a shared and VPS and have lost count how many times you guys have been down!!!!!

And pleeease to all those DH suck-up, fanboys… dont even bother sticking up for them.

This has become a pathetic excuse for a hosting company. As soon as i can get my data in order i am moving on and will NEVER recommend this unreliable host to anyone!

All my sites, All my clients sites, HTTP, FTP and E-Mail.

is MYSQL snapper server affected?

All my sites are down…

server is dabih

This reminds me of the olden days of Dreamhost, I think you are overpriced and under performing, can’t wait to leave!

Unreal, last week I had database issues, and now another outage. Are 12 year olds running this company or what.

Well, this is just peachy. ALL of my websites are down, no domain email, no FTP. I think we should all receive a credit toward our web hosting costs for these outages. Seriously.

geez, I can’t believe this, update my site send out the link on the social machines and boom, no one can see it. is it just me or does DH always go down at the worst times?

Im getting tired of spending almost $200 a month and getting nothing but downtime

Is down too….

Willa is down.

I was dumb enough to clear my cloudflare cache so my site is now displaying a cloudflare error message :S

I would say this has become all too frequent and it is disturbing your customers

mine is down… seriously how is it even acceptable to have this many issues. My term is up next month and I need to find a new hosting service. This is total BS

I just coming to the office and will start working. All my clients website are down likewise my office email :(. Just waiting some calling from our clients now :((

Yay! Here we go again! Hopefully it’s not another multi-day outage!

Way too frequent. Anyone can recommend a more reliable to migrate to?

Down down down, all the sites are down.

I do love the transparency of DH (this support blog and DH telling us when things are f**ked) but I think it’s about time to jump ship. Not just jumping at this either as I have been a customer on my personal account since 2002 and on a joint account with a partner since 2000. Old school. But I’m about to graduate. Thanks for all the fish!

2 different sites on 2 different servers down: chisinau and hydrobius. At least they finally posted something about it, I opened a ticket over a half hour ago

Ug… down again. Dreamhost, you really are not getting any better are you? This will accelerate our move to Rackspace.

Dreamhost, change your name to Downhost because you guys are always down or may be Problemhost or Worsthost. You guys are the worst. Your administrators should go and get some training to operate a data center. I believe you guys will be top performer in host maximum downtime.

happening way too often. anyone can recommend a more reliable host to migrate to?

365 days a year and this is the first day *any* of my web sites are impacted…

How’s that new and improved super redundant system working out? Not very well I’m afraid.

Pretty sad.

LYNX has been down for about 15 minutes. Good luck DH tech support – hope you had enough cups of coffee to bring your “A”-game to the end of this manic Monday!

This is frustrating on so many levels. Several outrages every now and then. Too frequent. I should start looking for another host when mine expires next March!!

I am googling hosting companies right now. I am over this

The funny thing is, I have five out of the six domains I have running on my VPS down. The main one is still up and running. Sounds to me like more of a router issue.

This has become a pathetic excuse for a hosting company. As soon as i can get my data in order i am moving on and will NEVER recommend this unreliable host to anyone!

Hey, you have fun with that. Don’t let the door hit ya were the lord split ya, Jimbo. When you find that more reliable host that never goes down, you come on back here and share. But be sure to share how much you have to spend to get that kind of reliability.

You see, I’ve been there, done that. At this price they all suck. Some a lot more than this. Sure, I could spend three times as much as I do right now to have a more “reliable” host.

Bottom line is, you get what you pay for. If you insist on doing it on the cheap, then this is what you get. Plain and simple. Want something more reliable? Try Rackspace or SPRY. Just expect to pay more.

Not being a “DH fanboy” about this, just laying down the simple facts.

I was just down two weeks ago for a significant amount of time. Please please PLEASE get us back up and running!! This is killing me!

Its going to be an hour now, my sites are down from 5’O clock PDT. :(

SIGH. I was on track to set a record for views today. Put a TON of promotion into a post today. Time wasted.

Anne H. – Dreamhost hosting’s monthly cost is $9.95. In month approximatelly 731 hour, that mean cost of hosting is 1.36 cent per hour. For 3 hour outage you can get only 4.1 cent credit (41 cent if you have $100 VPS) :)

Is completely down. very bad timing… Fix it please…

BIG SURPRISE, DREAMHOST IS DOWN?

Unbelievable services at dreamhost. I guess Dreamhost has been down more than being up. Please invest more money in your infrastructure.

how the DREAMHOST web site never goes down while our stuff is routinely taken off the air?

Crater is down
Sandtiger is down

Please help

DOEN DOWN DOWWNNNN

kokopu:hachigian is down :( all websites, ftp, mysql is down.

Your only job is to keep the servers going, you are a hosting company and can’t even host on a regular basis. Always some excuse. I never should have switched. I can honestly say that the last hosting company had 2 downtimes in 3 years, and one of those was during hurricane katrina.

well, at least Dreamhost has a status update, unlike many servers, who just leave you hanging with no knowledge of what’s going on. good luck getting it fixed… please.

I disagree with the calculation regarding the cost of downtime. For ANY business it is measured as an opportunity cost. let’s say we would have a group legal action against Dreamhost we would add up the cost of lost business because of their not keeping up the service up 100% especially during business hours.

All of my sites have been down for about 35 mins. Server is Gianfar MySQL is Morwong. No ftp access either. Under my Tracked Outages, I got back this response: “Outage resolved: No server-wide problem was found” ….Um, there’s a problem, I have 20+ sites down.

Haters. Things break. Do you know what happens at other hosting companies? Things go down and people complain just like here. Want a server that never goes down? Build it yourself. Contract the best nerds you can find to build you one for a LOT of money and then guess what? It will go down too.

And we’re back! Yay! :D

2 Dedi’s just came back online. Thank goodness!

My site been up 97% of the time. Don’t know why other are QQing when stuff happens. That life. xD

Any updated about this f**ked up service?

Awesome! It’s back. Fast work guys. Thanks so much :)

I’m back! Thanks DH!

Gah. Bunch of whiners need to dial it back a notch, eh. 365 days a year and you’re down for a matter of minutes? Jeez.

I’d be interested to know if this downtime had anything to do with the network changes last friday. Can you guys inform us on that? Thanks

I’m backup as well.

I have been with you for years now, and I’ve been very happy until recently. I understand you have probably been expanding faster than your infrastructure can be upgraded, which is fine until it starts interrupting the customer’s services. I did not renew my 2 year agreement because of the downtime issues. I do hope your services become more reliable in the next few months because I would rather not switch hosting companies. It is very difficult for me to run a business when our services keep going down.

Half of my sites completely down, and just the ones I need to update now… :(

arno kroner – I have bad news for you: “DreamHost is under no obligation to compensate Customer for downtime, whether the downtime be caused by Customer, DreamHost Web Hosting, or DreamHost’s upstream providers.”

Actually 1 hour outage per month is not too bad, it is only 99.9% uptime.

Thanks for Tech Support.

My sites are up…for now. Hope this lasts.

@DJ ALLAN stop talking BS out of your B#M , dude i pay top dollar for my VPS DH. I have accounts with OTHER VPS and not a single downtime for over 1year now. With DH i have been here 6or so years and each year they get worse and worse and worse and worse and worse.

You are like one of this dudes who just loves to stick up for something even if its S##T

Get real! and read the posts, im not the only unhappy top paying client!

@SCC you too p##s o#f

Nice mine is back…laggy but back it seems…Thanks!

Sorry but I agree with everyone else who is frustrated right now and is sick of it. I can see that a lot of people have multiple sites and I couldn’t imagine having more than just the one that I have. Downtime can hurt more than some may realize. I understand that some of you may not have had many problems and I’m glad you haven’t.

I understand things happen and I could deal with maybe only a couple times during a whole year but in the last TWO MONTHS my site has been down SIX times for 12-24 hours at a time. That’s beyond ridiculous and like I said, can be very damaging especially when your site becomes unreliable to it’s customers and traffic.

I have 6 months left and then I am most likely moving on – reliablity is a must. Already researching other hosting companies.

Back up! Thanks guys :)

can anyone recommend a good alternative to DH?

Funny. Says it’s resolved and my site is still down. Go figure.

My site is up! Why? Because I host it at Linode.com! Yep! I bought my domain at Dreamhost, I used the VPS/MySQL servers, and for the past two years nothing but trouble. I moved my domains AND DNS to Linode.com where I manage all of my sites and have NEVER had a problem. If you care about your site, and need the hands-on of a VPS, then Linode.com is the way to go. Afterall, wonder where the site you’re reading this comment on is hosted? Linode.com.

In all fairness, I still host my personal sites with Dreamhost because they are not important as my business site.

@Nicole, i agree totally. And quite frankly im annoyed with those who are annoyed at people getting annoyed.

I have used KnownHost for one client who has over 20 domains and its impeccable, the support is almost instant ( within minutes ) and they will do anything for you! literally help you install extras into your VPS help you set up SSL cert’s.

I do NOT work for them but frankly I have to say that as soon as i can get my data and email accounts all sorted and listed i’ll be switching the flick on the DNS from these guys.

wwwquickonlinetipscom/archives/2011/07/knownhost-vps-hosting-review/

July 30th, 2012 at 6:23 pmdomainleasehold Says:

negative comments at 5:30pm
glad comments at 6:00pm
For $20 per month you are getting a great service from these people. If your hosting is that mission critical, pay for a more robust service – or host in a different time zone then the customers are sleeping while all this IT crap goes on !!

Everything is working fine now, I didnt even notice it.

http://www.colombio.com

@domainleasehold

three letters for you LOL

frankly i dont care if my sites are back up, i pay for VPS and a fair bit fo ram. My monthly bill is the same as if i was to choose Knowhost VPS which are 100 times more reliable and support actually answer you. So you just keep on being a little DH fanboy and sucking up until one day you get sick of it too and realize how bad these guys really ARE!!!!!!!!!!!!!!!!!!

my sites are not back…

i just ask for a list of servers affected to resolve future actions….

@DJ ALLAN stop talking BS out of your B#M , dude i pay top dollar for my VPS DH. I have accounts with OTHER VPS and not a single downtime for over 1year now. With DH i have been here 6or so years and each year they get worse and worse and worse and worse and worse.

I pay $200.00 a month for MY VPS. I am maxed out. That isn’t really the point. My uptime is still around 99.9 percent with the occasional drop.

If you have VPS accounts that hosts that never go down, then why in the blue blazes are you sitting here complaining? Get jiggy with it and move on. Don’t just sit there and complain about it. Nobody is stopping you. Honestly, threatening to do it isn’t going to make Dreamhost shudder in their shoes — hell, they don’t even bother reading this stuff here.

I’ve never understood the people who come here whenever their sites are down and then make the threats that they are going to move. Do it already! Why bother venting about it here — is it going to make your site come up faster?

For that matter, all those people who have “clients” or their entire livelihood is riding on having their websites up and running — this is the wrong place for you. If I was one of your “clients” I would seriously question your judgment.

I pay for the service I need, and I get what I pay for here. For the most part it IS reliable, save for the rare occasion that it goes down. When it does go down, there is nothing I am hosting here that is all that critical that I start sweating bullets if it is down. I have a service for the two sites that I DO require such reliability and … wait for it… I pay much more for that reliability.

Get real! and read the posts, im not the only unhappy top paying client!

I have — for several years. Each time a server goes down, each time there is a power failure, or a router goes out, there are the predictable comments like yours. Did you think you were unique? That you are the first to start googling for new hosts, or threatening to take your crap and move? Or just beat your chest and complain how bad Dreamhost is?

What is funny is that every time I do wind up coming here, I see some of the same people making the same empty threats to leave — yet they are still here. Why? Because for the most part, their stuff stays up and running, and on the rare occasions something happens, it is suddenly the end of the world as they know it. But once things are up and running again — they stick around. Until the next time.

So if you are going to leave, then leave. Sitting here responding to MY comments is time that you could be spending moving your site(s). Don’t let me hold you up…

However, I don’t think you will move. So, see you next time things go down.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Power Issues in Los Angeles Datacenter (RESOLVED)
6:03 am

Power Issues in Los Angeles Datacenter (RESOLVED)

Posted (July 30th, 2012 at 6:03 am PST) by chalise

We are currently experiencing issues with power strips in two of our rows in one of our Los Angeles datacenters. This is affecting a subset of our shared and VPS host servers. Our team is currently looking into this and working to restore connectivity as quickly as possible. We will update this post with further information as it becomes available.

Update July 30th, 2012, 6:15am PDT: Our datacenter team was able to locate and resolve the issue. All sites and services should be back up and running now! Thanks for your patience.

This entry was posted on Monday, July 30th, 2012 at 6:03 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

4 Responses to “Power Issues in Los Angeles Datacenter (RESOLVED)”

All sites are down.

All three of my websites are down. It would have been nice if they listed the actual web server names that are affected with this power strip issue. I don’t know if this issue is the reason my websites are down or if it is coincidentally another reason?

More info is better!

My webshop (on Monrovia) is down on a really important day. Sods law!

@Dave Thomas: Sorry to hear that. We don’t host wendys.com.

@Michael G: I was just compiling a list of servers when the issue was fixed. I agree, that more information is good. You are likely on monrovia and would have been affected.

@Ben: Your site is back up, it was affected by this issue.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – Finalizing Core Upgrade (Los Angeles Data Center) – Friday July 27th 8pm PDT (6 hours) (RESOLVED)
5:17 pm

Network Improvement Series Maintenance – Finalizing Core Upgrade (Los Angeles Data Center) – Friday July 27th 8pm PDT (6 hours) (RESOLVED)

Posted (July 25th, 2012 at 5:17 pm PST) by Ralph

Our Network Improvement Series Maintenance will continue on Friday with the cut over of the physical connectivity switch from the old hardware to the new. We are excited to get this completed. This step is the last major step in our Network Improvement Series Maintenance. We are seeing improved performance and network reliability as a result of the Series Maintenance though we still have some optimization to complete.

The maintenance on Friday will switch over specific subsets of our data center. This means that the move will happen in small chunks rather than the full data center at one time to minimize the impact on live services. In our previous upgrade we were able to limit downtime to these subsets to under 2 minutes each. If service interruption for a specific subset exceeds 5 minutes we will roll back the changes.

When the maintenance starts we will be monitoring your comments and providing you with periodic updates as we progress through our process.

Update Saturday July 28th 1:50am PDT: At this time, we’re concluding the maintenance window. We didn’t get to everything we’d planned on, however we were able to take care of about 70% of the work. We did this with very little service interruption over the entire 6 hour window. At this time, we’ll make a separate announcement, detailing our plans to finish up the work we’ve yet to complete. We thank you for your patience during this window!

Update Saturday July 28th 1:15am PDT: We are currently experiencing problems with a row of mysql servers. Some downtime may be occurring, and we apologize for this. We’re working on restoring service ASAP.

Update Saturday July 28th 12:45am PDT: In an effort to troubleshoot the problems we had earlier, we’re rebooting a set of network hardware on non-customer facing gear. If this goes according to plan, and solves an issue we’re seeing, we’ll be able to reboot the customer facing hardware and hopefully fix the problems we saw earlier.

Update Saturday July 28th 12:13am PDT: Moving on to another row, we aren’t anticipating any issues to pop up. Previous non-customer facing services went as expected.

Update Saturday July 28th 12:00am PDT: We will be extending this maintenance another 2 hours till 2am PDT. Everything is running smoothly with the exception of one rack earlier. We are working on that rack in preparation for another try once we complete the rest of the data center.

Update Friday July 27th 11:45pm PDT: We are moving a group of non-customer facing rows at this time. Everything is currently going as expected.

Update Friday July 27th 11:20pm PDT: Moving on to the 6th of 11 rows. Previous row was true to form and completed with minimal to no impact.

Update Friday July 27th 11:00pm PDT: Previous row completed and we are happy to report, that we’re continuing with negligible service interruptions. We are moving on to the next largest row.

Update Friday July 27th 10:50pm PDT: We are in the process of moving the 4th out of 11 rows. The current row is one of the largest in the data center.

Update Friday July 27th 10:30pm PDT: Another successful rack moved with unnoticeable service interruption. We are moving on to the next switch.

Update Friday July 27th 10:15pm PDT: The reattempt on the switch was unsuccessful. We are moving on to a different switch and will come back to this previous one later.

Update Friday July 27th 9:45pm PDT: We are reattempting the previous switch now.

Update Friday July 27th 9:35pm PDT: Correction: Previous switch was still being worked on and had some issues. We have rolled back the change to troubleshoot in a non-production environment in order to reduce customer impact.

Update Friday July 27th 9:25pm PDT: We are moving on to the next part of the upgrade. The previous switch was upgraded without any noticeable service issues.

Update Friday July 27th 9:00pm PDT: This next switch was not customer facing. It is now successfully moved. We are now moving on to the next switch.

Update Friday July 27th 8:45pm PDT: First row has completed successfully. We are now moving on to the next switch. There was less than 2 minutes of service outage on the first row we just completed.

Update Friday July 27th 8:30pm PDT: We are still working through the first row and have thus far seen only slight and quick service interruptions. We are in the process of testing this row to make sure everything completed successfully.

Update Friday July 27th 8:15pm PDT: We are now starting with the first row of machines. Machines on this row will experience a short service interruption.

Update Friday July 27th 8:00pm PDT: Physical cabling is underway as we prepare to start on the first set of machines. To this point there has been no service downtime.

Update Friday July 27th 7:50pm PDT: Our team is assembled and getting ready to start on this maintenance. As we progress through the maintenance we will provide periodic updates and assist you both through your comments and through our support system.

This entry was posted on Wednesday, July 25th, 2012 at 5:17 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses to “Network Improvement Series Maintenance – Finalizing Core Upgrade (Los Angeles Data Center) – Friday July 27th 8pm PDT (6 hours) (RESOLVED)”

My site has been down since this started

@ Thomas Holland,

You’re not actually in the data center we’re working on, so submit a support request, and our techs will look into it for you!

I found that the ip is loading and I am getting this error: http://wiki.dreamhost.com/Bad_httpd_conf I will contact support.

All my sites are still not reachable. I can’t believe we weren’t notified that our sites could have downtime. :(

Brian, you are a very bastard man, and incompetent!

Confirm down: all my sites unrechables, and support not reply at any of my requerst! Incredible!

We haven’t been getting e-mails since this started. We were expecting entrees to our contest and now have to postpone it a whole week since we “didn’t reach our threshold”. Turns out we weren’t getting our e-mails this whole time and now have no idea how many people actually signed up. Why were we not notified personally prior to this maintenance?

My site has been down since Thursday. This is getting ridiculous… This is effecting my business… Not cool.

Our sites have been down for two days now, the fourth separate occasion this month. There is never any notice as to when you plan on taking down our servers other than posting on this page which is obviously an ineffective way of notifying us because it requires us to monitor it 24/7. To make matters worse you don’t even know which sites are on which servers.

You hide behind the fact that your service is cheap and therefore substandard support should be tolerated. The problem with this is that being considerate costs you nothing and people also know that. Most people don’t expect 100% uptime; all they expect is notice of planned downtime so they can work around it.

A) Work out which sites are on which servers.
B) Email those clients in advance of any planned downtime. You have our email addresses so please use them.

That is all you have to do. So simple.

Our e-mails have been down ever since and no one can register for our site because the activation mail won’t send. I’m deeply upset by this.

Site down since yesterday, any updates on when this will be fixed?

My HTTP service is timing out. Mail and FTP seem to be working. I thought this downtime was over?

nevermind.. it’s back up. yay!

We still can’t access to our host website and service since yesterday. Any update with this?

All my sites on my VPS are down, again! This is so bad, can you believe the sites on Dreamhost shared cheap hosting are ok! Please give us an update, clients will start calling any minute, I am in the UK.

For how long has my site been down? And where is your WE support team? What about all the upgrades you wrote about in your newsletter? This is terrible, again.

My site has been down for a long time. Any update when will it start working? Is somebody looking at it?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 11am PDT (2 hrs) July 26th (RESOLVED)
12:06 pm

Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 11am PDT (2 hrs) July 26th (RESOLVED)

Posted (July 25th, 2012 at 12:06 pm PST) by andrea

This Thursday, July 26th, we will be installing a backup 10 gigabit internet feed in our west-coast datacenters. This installation will be adding a new internet feed in both locations. This will further improve redundancy on our primary 10 gigabit lines in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the great news of further data center redundancy.

This entry was posted on Wednesday, July 25th, 2012 at 12:06 pm and is filed under Data Center Maintenance, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 10am PDT (2 hrs) July 25th (POSTPONED)
2:55 pm

Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 10am PDT (2 hrs) July 25th (POSTPONED)

Posted (July 23rd, 2012 at 2:55 pm PST) by andrea

This Wednesday, July 25th, we will be installing a backup 10 gigabit internet feed in our west-coast datacenters. This installation will be adding a new internet feed in both locations. This will further improve redundancy on our primary 10 gigabit lines in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the great news of further data center redundancy.

Update July 23, 11:42am PDT: Unfortunately, we were unable to finalize some of the necessary pre-testing for this maintenance. We are rescheduling for tomorrow morning, see new status post above. As a reminder, there is no anticipated impact to your services.

This entry was posted on Monday, July 23rd, 2012 at 2:55 pm and is filed under Data Center Maintenance, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Issue in Irvine Datacenter (RESOLVED)
7:56 am

Network Issue in Irvine Datacenter (RESOLVED)

Posted (July 23rd, 2012 at 7:56 am PST) by Justin@DH

We are currently experiencing network connectivity issues with our internal private network of our irvine data center. The internal network is responsible for dns and configuration of services from our control panel. We are experiencing issues on some but not all shared mysql services as a result of the dns issues. This is resulting in site downtime for a small subset of our dedicated, VPS and shared hosting customers. Our network engineers and system administration team are working to restore connectivity at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.

Update July 23, 9:30am PDT: All services have been reporting OK for the last 40 minutes, so we are going to go ahead and resolve this post. The cause of the outage, a specific DDoS attack, has been found and will be analyzed for better protection from this newer kind of attack. If you are still seeing any issues with your service, it may be unrelated to this issue so please contact support and we would be happy to help.

Update July 23, 9:00am PDT: We are still watching the situation, but we believe the issue is resolved. We are checking the reported sites from customers to make sure all is resolved before this post is resolved. We will update this post again shortly.

Update July 23, 8:50am PDT: The work is continuing on resolving this issue. It does appear to be similar to a previous network related downtime in a different datacenter that was ultimately due to a DDoS attack. We are investigating all avenues and that possibility has not been ruled out at this time. We will update this post again shortly.

Update July 23, 8:26am PDT: We have the entire network and admin team examining the situation at this time. The issue appears to be related to the firewall pair of the specific datacenter. We are continuing our work on restoring service and will update this post again shortly.

This entry was posted on Monday, July 23rd, 2012 at 7:56 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

29 Responses to “Network Issue in Irvine Datacenter (RESOLVED)”

(*&#%^$! I got no one to blame but myself :-(
We went through major issues a short while back, should’ve left then, but believed they crap about major upgrades…will never happen again…blah blah blah – and stayed put. So stupid.

That’s ironic timing. I was in the middle of tweaking my configuration to cut down on memory usage and thought I broke my server. No, you did.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Internal Server Error on MGM sites

Interesting… Right after you announce that you completed major upgrades? Why don’t I EVER have these problems with the other hosting companies I use?

So are you guys experiencing slow as heck sites today as well? Our sites don’t load up at all or are so slow. Is that the reason? We are not completely down but slow as heck

My site (shop) is not working, cannot access to my server via FTP.

All my sites are stuck with #1040 – Too many connections. My customers are mad at me…

Count me in as another of the “few” who are down yet again… Wonder if there’s any way to get transferred out of this Irvine location other than to just leave Dreamhost. Seems like they have nothing but trouble there that they aren’t having at the other one.

Thanks for the announcement. I thought that I had done something wrong when I couldn’t access my site this morning.
I got this: Error establishing a database connection
Looking forward to going back to normal. I’m sure that you guys are working hard on this.
Thanks.

might need to pause Google adwords to save some money. Not good

How do I know which data center my sites are located in? Having slow site issues, but nothing down…

July 23rd, 2012 at 8:27 amBreno Assunçao Says:

my site is back just now!! i can access external netwoks again. try yours

A new big client is very mad at us for this… We have a promotion running in their site… This could mean we’re gonna lose the account!! @#$%¿!?@#$ :@!!!!

My site is back to normal. Thanks a lot.

ERROR: Could not complete request.
Query: SELECT “INBOX.Sent”
Reason Given: [SERVERBUG] Internal error occurred. Refer to server log for more information. [2012-07-23 08:39:22]

It happens with some accounts. Sometimes on INBOX or TRASH folder

Is there anyway to switch off of the irvine servers? It seems they are the most problematic. Do you offer other data centers for dedicated servers?

@Rob: We do apologize for the issues here, and we are working on getting them resolved ASAP.

@dwnomad & @Renato S. Yamane & @Francisco: Please try again, your sites were loading when I checked it at this time.

@Cyrus: It is possible that if you are on an affected mysql like octobass or cello, then it could be related.

@VK: That would be a related issue to this, there are around 5 affected mysql services out of 100’s that are not working properly due to this. We are working on getting them working ASAP.

@Elsita: We are indeed working hard on this. The site you linked appears to be loading at this time.

@Jacob: If you let me know what webserver you are on I would be happy to check for you.

@Breno & @Elsita: Some service has been restored, but not fully, so we are continuing our work.

@Daniel Romero: That issue would be unrelated to any network outage, please contact support and we would be happy to help.

@Jeff: Sorry but this is the only datacenter currently for dedicated servers.

Still no access to my dedicated server, unfortunately…

One of my websites is working again, the other is still having problems. Will this be because of a slow response from your fix to the network issue?

@Jeremiah: I cannot find any open tickets with your name and your emails domain isn’t hosted with us. I can take a look but need additional info, either the dedicated server name or a domain, or the ticket number you submitted.

@Harry: That site was not down due to the network issue, but wrong configuration in your config file. I’ll write you with the details.

Asked via Live Chat around 7:45am & they told me there was some connectivity issues they were working on, so I didn’t bother opening a ticket. ds3642 is the server; can you tell if it’s a related issue or if I should open a support ticket?

Working fine on last 40min.

Thanks!

Back up and running, thanks!

@Jeremiah: It doesn’t seem related, I got it up for you right now.

Awesome, that saves my day. Thanks!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – Central Service Router Upgrade Monday July 23 – 8pm PDT(4 hours) (RESOLVED)
9:51 pm

Network Improvement Series Maintenance – Central Service Router Upgrade Monday July 23 – 8pm PDT(4 hours) (RESOLVED)

Posted (July 19th, 2012 at 9:51 pm PST) by Brian@dreamhost

On Monday, July 23rd, we’ll be upgrading the network hardware that is responsible for our central web panel (https://panel.dreamhost.com) and all web based email service (http://webmail.dreamhost.com and http://webmail.yourowndomain.com) as well as our sign up and contact forms.

Related to the outbound DDOS that affected the firewall pair at one of our Southern California datacenters last week, we will be upgrading that infrastructure to be able to better handle spikes in traffic in the future. We will be performing as much of the configuration staging as possible before the physical cutover so there should be very little disruption outside of that brief period.

We anticipate a 4 hour window, and expect some downtime for the afore-mentioned services. Rest assured that website uptime, standard email delivery and IMAP/POP/SMTP will not be affected by this window.

If you have any questions or concerns, please contact support with the details, so we can assist you!

Update July 23, 11pm PDT: We have successfully completed the maintenance, quicker than expected and with no significant downtime. Our central services are completely transferred over to the new hardware which has improved specs and redundancy.

We now consider this case resolved. If you have any questions or concerns, please contact support with the details, so we can assist you!

Update July 23, 10pm PDT: We are continuing to work on the backup hardware, still without significant service interruptions. We will update this post within the hour or sooner if a status change necessitates it.
If you have any questions or concerns, please contact support with the details, so we can assist you!

Update July 23, 9pm PDT: The maintenance is progressing as expected, without significant service interruptions. We will update this post within the hour or sooner if a status change necessitates it.

Update July 23, 8:28pm PDT: We successfully transferred over to the upgraded hardware. We are now moving on to the backup hardware. All services are up and working correctly. We’ll continue updating as we progress with this maintenance.

Update July 23, 8:10pm PDT: We are starting the maintenance as scheduled. We’re anticipating minimal downtime to services related to the DreamHost panel, webmail, signup and contact forms. We apologize for the inconvenience the We will post periodic updates as we progress.

This entry was posted on Thursday, July 19th, 2012 at 9:51 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Network Improvement Series Maintenance – Central Service Router Upgrade Monday July 23 – 8pm PDT(4 hours) (RESOLVED)”

Could you please update the MediaWiki Installers?

Thanks.

I’m a dreamhost customer and it’s a fair question. Oh, and Brian, why don’t you learn English sometime in between your useless ranting mmmk? Or are you illiterate too? Just wondering.

My my my. Such language. Thin skinned are we? Why don’t you just wander off and troll elsewhere? Or are you here because you have nothing better to do? Oh, that’s right, you don’t have anything else to do. I wonder when you’ll give up and find somewhere else to show the net how witless you are. Enjoy your sad little life fake Brian. Don’t let the door hit you on the way out.

Brian H, what do manage at Dreamhost? A group of incompetents, who tries, in vain, to run a nightmare of undersized and legacy datacenters?

So Brian H, I´m sorry for being a incompetent user… I´m really sorry because I´m pretty sure that my web site made hundreds of users to be outaged last week. Sorry because it´s clear to me that some command I gave in your server made us all loose some data in a way that you holly competent data center managers were not able to recover.

Really dude, I´m so sorry about all that.

@Ricardo: I work for Dreamhost and can confirm Brian H is a support manager here at Dreamhost, however, this is not Brian H writing in the comments. I have deleted the spammer comments and can assure you no one working at Dreamhost would write these things. The spammer using Brian’s name was using a fake email address that is not a support address or a valid Dreamhost address. If you have any questions or concerns about our service or site, please write into support from the panel. I have disabled the comments on this post for the time being.

@Wondering: Regarding MediaWiki, we’re working on the upgrades and should be coming out soon.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Upgrade (Orange County Data Center) – Redundant Internet Feed Upgrade – 10am PDT July 13th (POSTPONED)
8:03 pm

Network Upgrade (Orange County Data Center) – Redundant Internet Feed Upgrade – 10am PDT July 13th (POSTPONED)

Posted (July 12th, 2012 at 8:03 pm PST) by Ralph

This Friday, July 13th we will be installing a backup 10 gigabit internet feed. This installation will be adding an additional new internet feed to the Orange County data center. The reason for this is to further improve redundancy on our primary 10 gigabit line in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the news of further data center redundancy.

Update July 13, 4:09pm PDT: We’ve had to postpone this upgrade to our network due to peering issues on our provider’s side of things. We’ll create another post, when this has been rescheduled. –Brian H

This entry was posted on Thursday, July 12th, 2012 at 8:03 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #5: Core Router Upgrade (Los Angeles) Monday, July 16 – 4pm PDT (8 hours) (RESOLVED)
3:58 pm

Network Improvement Series Maintenance #5: Core Router Upgrade (Los Angeles) Monday, July 16 – 4pm PDT (8 hours) (RESOLVED)

Posted (July 12th, 2012 at 3:58 pm PST) by Ralph

In the past month, we have made significant progress with our network upgrade project. Due to these successes and our ability to minimize downtime we are confident that we can proceed with the next step of upgrading our core router at our Los Angeles-based data center.

The successful Orange County upgrade helped us find a way to avoid affecting customer services as much as possible. Since our confidence is high, and we anticipate no material customer impact we are starting our maintenance window a bit earlier than usual, at 4pm PDT.

The window will last 8 hours, until midnight. The Los Angeles data center is larger than Orange County, but the changes are the same and our new streamlined process has been extremely effective.

During the maintenance, we will continue to update you on the progress we are making. If you are experiencing problems prior to this maintenance window please contact support as your issue will not be related. During the window we will be monitoring support messages and status comments to ensure that customer impact is limited as much as possible.

Update July 17th, 12:05am PDT: This network maintenance is now successfully completed. Overall, everything went well, however, we did have 2 hours of spotty or intermittent mysql coverage for a group of servers. That should now be resolved, so there should be no lingering issues from this maintenance window. We appreciate your patience with this, if you are experiencing any issues, please contact support with the details, so we can assist you.

Update July 16th, 11:15pm PDT We’re nearing the end here, just tying up some of the lingering loose ends on our backend. There should be no service outages related to this maintenance at this time.

Update July 16th, 9:45pm PDT We narrowed the issue down to a specific row of servers. After some quick troubleshooting, we were able to restore service to that row of machines. At this time, we’re not seeing anymore problems. If you are, please let us know via the comments if you see any issues with database connectivity.

Update July 16th, 8:45pm PDT At this time, we’re seeing some intermittent connectivity issues between hosting servers (VPS included) and mysql servers (mysql VPS included here too). We’re working on the problem and will update as we get more info.

Update July 16th, 8:15pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. Several sections are now completely done. If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back for further status updates.

Update July 16th, 6:15pm PDT The maintenance has been progressing as planned so far. We are taking a slow and steady approach, and have yet to see any service disruption. If you are experiencing any problems, or downtime, it may or may not be related. Please contact support with the details, so we can assist you.

Update July 16th, 4:10pm PDT We are preparing to start the hardware change in our datacenter, to upgrade the core routers. Please check back here often, as we will be posting updates as we progress. As mentioned above, we expect short amounts of downtime as we change the hardware. We appreciate your patience during this process.

This entry was posted on Thursday, July 12th, 2012 at 3:58 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

31 Responses to “Network Improvement Series Maintenance #5: Core Router Upgrade (Los Angeles) Monday, July 16 – 4pm PDT (8 hours) (RESOLVED)”

You didn´t note us in advance.
You try to put it as it is an upgrade.
You don´t even give a reliable estimate.
It doesn´t matter if your service is cheap, this kind of thing is unbelievable!

I´m glad that I don´t put any critic service in your hands… you do not deserve our trust.

Ricardo:

Do you have a reading comprehension problem? The notice was posted here on 7/12. That’s a 4 day advance notice.

Not playing the ‘fan boy’, because DH has certainly been performing sub-par lately, but you need to stop playing with your computer and go back to school and learn some basic skills.

What always cracks me up about these notices, is we have no idea as customers what the “Los Angeles” VS any other data center actually means… which servers and/or services are housed there?

Do I expect some interruption, no interruptions or just partial interruption?

July 16th, 2012 at 7:11 pmit's getting better! (grin) Says:

“Do I expect some interruption, no interruptions or just partial interruption?…”

Yes.

mMleeC

I don´t play with my computer, I work on it. Actually I´m wasting my time right now answering you…

But, I don´t come here regularly to check if they posted something about my servers. I expect to be noted by EMAIL for something important like this.

As I said, I don´t host anything important in DH, I do it in more serious services like Joyent. There, they tell us in many ways that really reaches us, in advance, and usually they keep to their estimates.

Well, like I said… enought time wasted with you… pfff

well… psmysql7458 is offline/unreachable now (within the maintenance window). And it’s hard to put a ticket into the dreamhost panel.. it’s acting sketchy.

@Alan, you’re not related to the maintenance, I’m having a support staffer check into your issue. Also, what are you seeing on the web panel?

Brian, panel (and my credit card) shows me fully paid, billing contacted though. The panel earlier was acting flakey and was being unreachable for several minutes. It’s working for me now.

why is it every time you do an update, all of my sites go down. Do you just not like me or something? It always seems to happen when I do an email blast too.

Kevin, we love you, promise! Let me know what domain you’re having problems with, and I’ll check into wassup for ya!

My site started timing out about 90 minutes ago and is still not reachable.
In past incidents this was due to not restarting Phusion when the servers were brought back on line, but tonight it’s just timing out. Please help…

My VPS server went unresponsive at 8:51 – I was expecting to see more comments here! I’ve contacted support, but what are the odds it’d be just me?

@Rob I checked your site, and it seems to be up, are you still having problems?

@the other Kevin, the site you listed, deep fling.com is up and working, is that the site on your VPS?

kevinfocht.com
thecreativeorange.com
thebigrc.com
thebigsc.com
fochtcreativeclub.com

Just to name a few

All back up now. Only took a few hours.

Next time you do an update, make sure someone elses sites go down and not mine

Kevin

UNBELIEVABLE!

STILL DOWN

Our Sites have been down for about 24 hours now!! Still Down!

2 days downtime. In this time a decent hosting company would build an entire data center from ground up! SHAME ON YOU!

Fuk all faggot spammers!

please check mysql @ porbeagle — it’s down for hours! my websites are all down because of this!

Extremely slow responses, frequent time outs. Opened a support ticket 13 hours ago, still no response. Can sometimes get the website to come up, then can’t get to the admin pages. This is on 4 different websites. Am really frustrated by over 48 hours of little or no access to my sites.

why don’t you guys update this page and @DHStatus with the latest news about the porbeagle outage? the red message in DH’s control panel just shows the generic “wait up until backup is restored” is annoying and doesn’t help at all

They recovered my host, but the svn is outdated, 7 days outdated.
Incompetents.

And this status says it is solved… solved for whom?

My Sites are still down… :(

CG, I don´t know about you, but I am moving from DH. These guys suck way too much.

You guys on porbeagle that got 7 days outdated recovery are lucky. They told me they have NO backups of ANY of my databases. All my dreamhost sites that depended on a database are completely gone after their hardware failure. I guess they were backing up to the same box?

Also, they told me in an email that they have permanently lost hundreds of customer’s data from porbeagle. I would think that would warrant a post on dreamhoststatus, no?

A three day “outage” that turns into a permanent outage. I guess from Dreamhost’s perspective, that counts as all systems green.

DEAR DH,

Thanks for the 4 day notice of your upgrade project. If I wasn’t on vacation last weekend, I would have backed up my sites, since being out of the country makes this challenging. For a sensite upgrade like this (where something could go wrong – YEP, LOOKS LIKE SOMETHING DID), a little more WARNING and ADVANCE NOTICE would have been appreciated.

MY RESULTS OF THIS “UPGRADE”
2 Days completely down last weekend, 3 days w/ spotty access
All sites rolled back to last month’s snapshot
Loss of data – More than 10 GBs representing $3,000 (not including time to restore and patch)

WHATA GREAT SUCCESS. PLEASE PAT YOURSELVES ON THE BACK SOME MORE FOR ALL OF YOUR “SATISFIED” CUSTOMERS TO READ AND BE IMPRESSED.

E-P-I-C F-A-I-L

Ryan said it all!

Look at this page: http://dreamhost.com/about-us/the-dreamhost-team/

Those two idiots trying to play cool smart guys.

I prefer to host my sites somewhere where people wear suits and cologne and DO NOT LOOSE MY DATA!

SUCKERS!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » No connection to DreamHost panel (RESOLVED)
11:11 am

No connection to DreamHost panel (RESOLVED)

Posted (July 11th, 2012 at 11:11 am PST) by andrea

We are currently having connection issues to the DreamHost control panel and other central web-based services (contact form, sign up and Webmail) – the good news is that this means that customer websites are not impacted. Our teams are working on getting these working again, and we apologize for the inconvenience this is causing. We’ll update this post again shortly as soon as more information is available.

Update July 11, 6:40pm PDT: All issues related to this status post have been addressed so we are setting it to resolved.

Update July 11, 5:30pm PDT: The services that were affected by this issue (webmail and panel) have now been fully restored. We have identified the root cause as malicious outbound activity from one of our customers that caused a subset of our load balanced firewalls to fail due to high memory utilization. Troubleshooting and resolving the issue required a number of physical reboots and configuration changes, plus adding memory to the firewalls. We’ve taken short term steps to prevent this issue from recurring, and will be implementing a permanent fix within the next 48 hours. Thanks again for your continued patience during our efforts to resolve this issue. As a reminder, only webmail (not POP/IMAP mail), web panel access and our signup form were not available during this issue. Customer sites were not affected.

Update July 11, 4:55pm PDT: We have made good progress and are in the process of bringing up all services now. We’ll update this post again shortly with additional details. Thank you again for your patience!

Update July 11, 3:47pm PDT: Our troubleshooting efforts are continuing full force. We believe we have the issue narrowed down to our firewall pair. Unfortunately both the active and standby are experiencing similar issues so we’re investigating upgrade options as well as software configuration improvements. Thank you again for your continued patience and at this time we are still only experiencing service interruptions with the DreamHost web panel, the signup form, and both webmail clients. (RoundCube and Squirrel Mail).

As a reminder, customer sites and other non-central services are up and running. If you have any unrelated issues, please provide specifics in your comment below and we’ll be happy to assist to the best of our abilities.

Update July 11, 2:40pm PDT: We are continuing to work towards resolution. We believe we have narrowed down the issue caused by one piece of networking equipment. Just to reiterate, currently the DreamHost panel, the signup form and webmail (Squirrel Mail and RoundCube) are inaccessible. We apologize for the continued inconvenience.

Update July 11, 1:45pm PDT: We apologize for the continued downtime and inconvenience and are working out different approaches in an effort to avoid any downtime to any additional services. Currently, your sites should be loading, only the above mentioned services are unavailable. We realize that support ticketing isn’t currently accessible, so if you have any unrelated issues please provide the specifics in the comment section (yes, we know we normally advise against this). We’ll be happy to help as much as possible.

Update July 11, 1:40pm PDT: System Engineering and our network engineers are working with the entire management team towards a resolution to this outage. We apologize for the extended service interruption and thank you for your patience with us as we get this sorted out.

Update July 11, 12:40pm PDT: We apologize for the continued downtime and inconvenience. We are still working on the issue but cannot provide specific details just yet; however, we have continued to check other customer-facing services, and so far the outage has been limited to the central DreamHost Tools (Account Control Panel, Webmail and Sign up form). Thank you for your continued patience as we strive to resolve this issue for you.

Update July 11, 11:50am PDT: Unfortunately, the attempts we made so far only yielded partial progress. We continue working towards resolution. Until then, the DreamHost panel, Webmail and Signup form remain inaccessible. We apologize for the inconvenience.

Update July 11, 11:25am PDT: We continue working towards resolution. Until then, the DreamHost panel, Webmail and Signup form remain inaccessible. We apologize for the inconvenience.

This entry was posted on Wednesday, July 11th, 2012 at 11:11 am and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

397 Responses to “No connection to DreamHost panel (RESOLVED)”

Webmail = No connectio
PAnel = No connction

Fine

I think we’re done with dreamhost.

Money back and good bye

Webmail is also down!

Webmail is having issues as well. Been a lot of webmail troubles lately, I’m thinking I need to move off Dreamhost.

yesterday webmail, today webpanel………..

Well, it is nice that you are fixing it, but when I most need my webmail to be working… is down.

How come no mention of the servers that have been down for 36+ hours?

Why can’t these updates be a little more informative? Like, “the router is fried,” or “the gateway is completely overwhelmed,” or “the chimp that we keep in the server room just pis**d all over the place, so everything’s dead, and we’re planning to fire him at the end of the current billing cycle.” Yeah, a little more detail like that would help.

You can do it!

It seems funny, almost everyday there will be a problem.. Replace your old machines.. People are using Intel Technology with powerful SSD featured and so on, but these ppl still using old AMD type of stuff.. You demand us to pay immediately the money for hosting but when it is affecting the customers, no compensation at all.. if i am rich now, i will move to LIQUIDWEB HOSTING because they have the best machines on earth..

No email, no control panel… wonder if even FTP is still working.

Clients are chasing me to kill me… thanks.

Every week at least it seems there’s a problem. Come on guys this is unacceptable for what we pay.

@jim.. humorous comment but really what would you do with more detail?

@chris.. they only post single server issues in the panel now… not here… oh. nevermind the panel is down….

Thanks for looking into this. Can’t believe how fast you guys noticed the downtime. It’s understandable but still overly harsh (IMHO) that some of these commenters expect no computer glitches. Ever. And solving problems in less than a day is unacceptable. I dare any of them to be a computer technician and get machines working with 100% uptime. But anyway, thanks for your hard work, hope this gets resolved smoothly.

@abdullah “Verily, with hardship, there is relief” (Quran 94:6)

Abdullah can you see, http://www.liquidweb.com is down right now *LOL*

I think this is all part of DreamHost’s comeback strategy. They will shed thousands of accounts with crappy service. Then a few months later when the load is lower, and performance better, they will beg all their old customers to come back.

i’ve already started migrating to another host :)
dreamhost’s strength is in their low price, not in their service.

New trend with younger hosts: “pay as you grow”, where your hosting is free-as-in-beer (with no ad banners) until you scale it. i’ve been using one of those for months without any sort of outage, and hey – it’s way less expensive than dreamhost.

Thank you for the update.

I’ve started migrating all of my sites to another provider. These ongoing problems are just a joke and completely unacceptable.
Adios Dreamhost

hi, even some websites are down!

Nice guys, had a meeting with a VC fund about to seal the deal and my mail goes down. This is weak, second day. The webmail has been slow since I started using it a month ago and now the entire system is down. 12pm meeting hozed. Weak. Fix it.

@LakeRate: More detail would at least give me something concrete to tell my customers, rather than, “Yeah, well, the hosting provider just ****ed on itself again.” Everyone looks like a moron = everybody loses: DreamHost, me, my clients, their customers, their friends and family, the squid at Jo-Heng’s Teriyaki down the street, everyone.

right behind you volkswag.. last chance dh.
you guys are making me look stupid

webmail – down
panel – down
ftp – down
website – down

you should be renamed as – Downhost from dreamhost !!!!

Mail is NOT down, people – ONLY WEBMAIL!!! Jeez, you have access to a computer to check webmail but no mail application? A smartphone? I’ve been sending and receiving all day long between my Mac Mail and my iPhone… Just because webmail is down doesn’t mean you need to get your panties in a wad.

The panel, well… That’s a different story.

Thanks for the update

No host is perfect – but you will be hard pressed to find a better price/value than Dreamhost.

It’s outrageous that dreamhost are shutting down my email service.

Come on this is twice this week… Can’t you get these problems fixed? With this economy we can not afford to lose any customers or business or we are getting a refund and changing companies.

I’m losing two customers right now because of this issue. They turned off the phone in my face. They called that I’m not a professional guy. And you dreamhost, what you can do to me?

July 11th, 2012 at 11:45 amDick Johnson Says:

Whiners – go back to surfing porn, where it’s always UP!

@jim… doesn’t really tell the client anymore than “connection issues in the datacenter” does..

there is always rackspace… oh. I forgot they cost a lot more…

wedmail down too, come on yall

This seems to be happening more and more lately.

I’ll gladly leave my non critical websites hosted at Dreamhost, but it appears as though the ones more important to me and my clients will be moving to a VPS else where shortly.

I’ll get my panties in a bunch when this place fucks me in a client facing situation you fucking douchebag.

@luiz … I always tell customers upfront from the beginning that the level of service they will get is in line with what they will pay… and if they want better service I can provide it at higher cost (which is a true statement) I let the customer decide i the would like to pay more for more reliability or a lot less for service that might have issues more often.

I’m done! I’m going out of this bad service :/

I just dont carry anymore

Over 2 hours and no update?? Seriously, they can’t tell where the bandwidth is bottlenecked or communication is broken on their own network? I’m starting to wonder if DH doesn’t really own its servers, but is simply using someone else’s enterprise hosting service.

Any update on the Orangestad server? Thanks!

oh my gosh, get these Fricking servers up!!

Webmail = no access
Panel = no access
Wiki = bad gateway

WTF is wrong with you?

I guess you don’t care

Really come on we pay good money and your services to us all is getting worse and worse why on earth do your servers and system keep having issues. I thought you were supposed to be improving things not making them worse.

A pis**d of customer

Jim: DH is in PDT, so they actually updated 4 minutes before you said that they hadn’t updated in 2 hours.

Still frustrated, but they are clearly actively working on it…

Hey!, i need see my panel and again is down ?, mmmm, maybe do you send alerts before make changes on server.

Anyone know good hosting alternatives?

@jim I havent gotten an update at all, 38 hours in!

@Daniel A “glitch” is one thing, multiple servers being down and no replies from support is different. This is the second 24+ hour outage in 6 months for me and im sure many others. From the looks of it this one will be longer than 48 hours

They don’t have redundancy on MySQL servers?

For those complaining that you can’t access webmail, I have a workaround you are welcome to use.

Since I happen to have RoundCube installed on Dreamhost, it means ANY Dreamhost users may use it if they like to check their Dreamhost-ed email.

http://www.promail.ca

Just enter your full email address to login.

The system is a clean base install of RoundCube connected to Dreamhost locally via imap. There is no phishing or anything. Take my word for it or no, but we’re legit ;)

Regards,
Robbie Ferguson

Started service 2003 Ended: Today. Can’t take it anymore.

Our site is down.

Started service 2006 Ended: Today. Can’t take it anymore.

wow, this is getting bad. Outages are getting more wide spread. Makes me wonder about the real status of this web hosting company. Looks like they are struggling. Hate to think I will have to go through a change.

it seems several of my sites are down too

Instead of spending tens of thousands of dollars sponsoring events like WordCamps (http://2012.sf.wordcamp.org/2012/07/11/dreamhost-to-sponsor-wordcamp-san-francisco/) maybe you guys should spend some dough on NOT SUCKING. Then people might actually suggest you for a host.

#neverendingsuckiness

anybody’s site showing hacks? mine shows hacks in Seattle, WA but not in MN. CDNs messed up too?

I wish I could upvote Jim’s comment:

Why can’t these updates be a little more informative? Like, “the router is fried,” or “the gateway is completely overwhelmed,” or “the chimp that we keep in the server room just pis**d all over the place, so everything’s dead, and we’re planning to fire him at the end of the current billing cycle.” Yeah, a little more detail like that would help.

But seriously, get your shit together.

Our site is down too. You guys are having way too many outages of late. Get it together.

All sites are down. 30 minutes later no response.

This has got well beyond a joke. VPS host went on Monday and is still not fully back, this is the second time this has happened in 6 months. There is no status pages for these server outages as there use to be, no doubt to prevent people publically complaining.

My site is down too. WTF.

Updates like this:

“Update July 11, 11:50am PDT: Unfortunately, the attempts we have made so far have only yielded partial progress”

tell me you have no clue what’s wrong or how to fix it, and that you’re doing the equivalent of hitting the broken thing with a hammer until it works again. Details, please.

All our Dreamhost hosted sites are down too.

Perhaps, instead of throwing an “open bar” party in Boston next Monday, a better use of that cash would have been to purchase some REDUNDANCY (in hardware) and COMPETENCY (in staffing.

If you complainers are paying $150 a year for hosting I’m not sure what you expect. If you want new machines then go pay the $400/mo on Rackspace. Otherwise shut your yappers already.

All down! Fu*$(* mayday!

lol @Jeff, get a clue

C’mon, guys. This is happening way too often.

I just signed two new domains with you, and I’m wondering if that was a mistake.
It seems as though there’s some sort of outage every other week.

Please do something to reduce the frequency of your downtime. It’s getting out of hand.

Finally, a voice of reason. Thanks, Jeff!!

@jeff has your site been down over 36 hours? if not, shut up

@Jeff: They claim to have over 300,000 accounts. For an average $150/yr (low, I think), that comes to $45,000,000 annually. They can afford it.

Again, you get what you pay for. Don’t host mission critical sites on an economy host. It’s not rocket science.

@jeff, ignorance is bliss.

@Jeff, so you’re saying 36 hours down is acceptable for a cheap host?

If it’s a routing issue…where’s the redundancy? BGP? Something…anything?

8 router-0.hq.newdream.net (66.33.201.1) 196.920 ms 196.927 ms 196.943 ms
9 * * *
10 * * *
11 * * *
12 * * *

[pacifica]/var# dig -x 66.33.201.1

;; ANSWER SECTION:
1.201.33.66.in-addr.arpa. 262 IN PTR router-0.hq.newdream.net.

[pacifica]/var# dig -x 66.33.201.2

;; ANSWER SECTION:
2.201.33.66.in-addr.arpa. 237 IN PTR rampart-0.hq.newdream.net.

I have over 45 sites hosted here, I am well aware of Dreamhost and what they offer. There’s no ignorance here, just a positive connection with reality. This is why I also have an account elsewhere for the sites that need more.

btw, im paying $25+ a month

@Jeff: Those of us who have been with DreamHost for more than a few years know that they can do better than this. They used to be top-notch. It’s sad, really, to see them fall, and keep falling, like this.

@Jeff are you on crack?
It doesn’t matter the priority of your site. You pay for the announced uptime.

@Jim I’ve been with them for 7 years.
If things are really this bad then why are you still here? It sounds like you’re a glutton for punishment.

@jeff Your logic is as follows: You buy a TV you go home, you plug it in and it doesn’t work. You blame yourself for buying it at Best Buy. You decide you shouldn’t complain its your fault.

This happened a couple of years back too. Massive suckyness for a spell and then good.

A business plan that can cope with growing pains would be a Good Idea(tm)

July 11th, 2012 at 12:53 pminconveniencedButOK Says:

@Jeff
Seriously! It’s amazing to me how many people are crying bloody murder here! If your/your clients website is THAT important, perhaps you could budget a bit more than $10/mo for hosting! AND we’re only talking about the panel and webmail! Sheesh y’all act like they deleted your sites and transfered your domains to someone in Nigeria!

One of these days, I am going to seriously put something to the effect of the following on my answering machine.

Thank you for calling the DreamHost technical support hotline. Please listen carefully as our menu options have changed to better annoy you. To do “this”, press 1 now. To do “that”, press 2 now. To do “a third thing”, press 3 now. If you do not want to press 1, 2, or 3 now, press 4 now. To try and leave a message with somebody whose voice mailbox is full, press 5 now. To repeat these options, press 6 now. To hear more options, press 7 now. If you are tired of listening to these options, press 8 now. If you have already jumped off a cliff as a result of listening to these options…goodbye.

Okay, back to work! :-)

Thanks Robbie Ferguson! The http://www.promail.ca/ worked for me :) I was able to access the webmail accounts waiting for critical client notes.

I’m not happy about the continued outages, but you my friend, have actually provided a work-around solution until the problems are fixed.

Mariska

I think it’s the chimp.

@J no, my logic is as follows: If you buy the “Sorny” brand tv instead of the “Sony” then you blame yourself.

You’re hosting with “Sorny”.

@Brian Harvey: Now that you mention it I do seem to recall that somewhat. It’s funny how we forget the bad things. lol Hopefully this will pass too.

@jeff Dumb, so dumb. Where did you get your GED.

Our entire site is down – it’s not just the panel.

What is going on? Last month one server had TEN DAYS of downtime, now my site on the ‘oranjested’ server is down, there’s hundreds of sites down, right now, and have been for 47 hours… with NO MENTION OF IT ON HERE!

WHAT IS GOING ON?????????

@ inconveniencedButOK ” AND we’re only talking about the panel and webmail! Sheesh”

Wrong! my site and sites from at least two other servers have been down for 38+ hours now

@Jeff, Ya, not sure what you are trying to accomplish here. You are complaining about complaining. Then saying its a cheap host so the service should be bad? What?

@jeff is a troll

@jeff,
see the top 10 best website hosting

http://www.ipage.com $79.90 per year!!!!

@Jeff The other hosting providers are not any better than dreamhost (when dreamhost has there shit together) they just increase there price and you believe them. You are a poster child of corporations world wide. Create an invisible problem, then people will believe the problem. OMG CHEAP HOST FAILS!

Hey come on people, just look at the great job they’re doing keeping dreamhoststatus.com up and going for us, so that we have a place to rant and complain. I wonder who they host it with? ;)

Oh, and by the way, the Dreamhost forum is also currently down right now too.

Down, down, down DOWN!

Webmail down, cPanel down, our server is down, the dreamhost forum is down.

Everything is DOWN!

My car had a flat tire this morning. I sure its related to this outage.

This is seriously affecting my business!!!!

If anyone in the panel is reading this, and have a recommendation to switch carriers, or at least email for now, please do so!

Thanks.

@hadavoz lifehacker.com had a list of top 10 hosting providers. google it.

I can receive, but not send email.

TAKE ACTION!

For those who are upset about DreamHost’s continued lack of priority on their server/network problems…

For those who are outraged that DreamHost is spending thousands of dollars on self aggrandizement while their network flounders:

http://2012.sf.wordcamp.org/2012/07/11/dreamhost-to-sponsor-wordcamp-san-francisco/

Here’s what you can do: Let the San Francisco local news papers know what is happening by sending in a local news tip:

http://www.sfgate.com/chronicle-contacts/

http://www.sfexaminer.com/info/contact

Or, send DreamHosts’s public relations office a kindly worded email:

http://dreamhost.com/contact/

Have a happy DreamHost day! >:<

Thanks for the updates. Things happen. I use Outlook and my iPhone to check email, so I’m good to go, and my website is up. Just keep the updates coming. Helps a lot…

Dear sir or madam,
For two days now, we are unable to access webmail or even teh panel; teh following error message restricts us from gaining any access to the website,
” Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

Please look into it
Regards
Rajkumar

dreamhost.com is working, where is it hosted??

Glad it helped Mariska – some small bit of sunshine in an otherwise dark Dreamhost day :)

Wayne, read my note above if you’re upset about not being able to access webmail. Mariska confirmed that my provided workaround does indeed work.

Good luck.
Robbie

You are killing me.

DreamHost VPS host server homie-vserver157 down for +36 hrs. and no mention of the servers that have been down here on dreamhoststatus.com… Since about 24 hrs ticket support missing, now -since about 11 am today- the web panel down into addition… What’s up here on DH?!?

http://www.promail.ca/

thanks Robbie Ferguson!
Unfortunately, it doesn’t work for me.

I switched to Bluehost.com – much more reliable!

Webmail and Panel Down Again…. hate this problem. Dreamhost Team, HELP HELP… quick solutions

What the hell is goin on with DreamHost !!
Who is guilty for ignit’n fire !

Please, hurry up, i have a lot of work to do.

It looks like my sites are down also, except when CloudFlare manages to display a cahced copy.

July 11th, 2012 at 1:15 pmNewbie Admin Says:

Dear Newbie Admins:

Webmail is down, but IMAP isn’t.

You know, the phrase ‘Class Action Legal Action’ has come to mind. Gross negligence is, I believe, grounds.

FTP is not working either.

We have problems with your service so I hope a discount about this 2 or 3 days without a good services, I need you respond for this issue and the money back.

Regards

my site is down again. this happens at least once a month now. i mean seriously this is so dumb

today i use webftp to upload files, I create 2 mysql database so the cpanel was working fine for me until now so why peoples say 36h+

anyway, Been with dreamhost since year 2000 and in the past 12 years, how often it went down.. well for me, i can count this on one hand… so ill wait until it is repair because their services are excellent.. and if some decide to get another Host.. keep in mind the grass is not always green on the other side of the fence…

Keep the good Work dreamhost

Norm

wow – bad. had a client meeting to do a demo. it cost me.

Well, guess I’m going to have to push back a dead line if this keeps up. Would nice to at least be able get my files so I can set up a local server to work on.

Agree with Norm. one has to be patient.
a few of my sites are down as well :-/

We’re having trouble with one of our email addresses also. I see the panel is now unavailable so no way to try changing password or getting help. However our other emails and FTP are working fine.

Poor @Jeff getting re-trolled by the trolls. If your site’s been down 36 hours, you probably should be setting up your new hosting elsewhere rather than posting comments on http;//dreamhoststatus.com

And before anyone asks, the site dreamhoststatus.com is hosted on another provider’s network.

Been a long time account holder, supporter and fan of DreamHost web hosting (since early 2005)… but these outages and downtimes are becoming absolutely ridiculous! There is no possible way you can live up to your 99% guaranteed uptime.

Seriously considering a new web host (unfortunately).

I dont know if that is related, but Dreamhost got their outgoing mail servers listed in an escalation listing on Spamhaus. They host so many spammers that Spamhaus had to escalate. So maybe DH just cant cope with the whole spam traffic… Spamhaus link:

http://www.spamhaus.org/sbl/listings/dreamhost.com

Hacked by the Borg, resistance is futile…

Not just panel but all of my sites. I’ve lost track of the number of times this has happened in the past 2 months. Where’s my uptime guarantee if I can’t file a support claim?

NO PANEL!!!!!!!!!!!!!!!!

J, the article I found, http://lifehacker.com/5545568/five-best-personal-web-hosts, ranks DreamHost best of those named. I switched from 1&1 to DreamHost for better service and better price performance.

@Norm Some of the servers (homie-vserver157 for one) have been down for two days with almost no response from DreamHost. Apparently, individual server problems aren’t being reported at DreamHostStatus any more. Waiting to see how this all plays out…

Their followup, http://lifehacker.com/5912254/most-popular-web-hosting-company-dreamhost, showed DreamHost most popular. I can’t find the article you mentioned.

All my sites/database are operational, plus mail is working though outlook, phone, etc, only problem is panel and squirrelmail. Only reason I noticed this was down is because I needed to send an attachment through squirrelmail, don’t know how everyone else has so many problems. Come to think of it, this is the first issue in 5 years with dreamhost that has had any sort of major effect on me.

# /etc/init.d/mysqld stop
# /etc/init.d/mysqld start –skip-grant-tables

?????????????????????????????????? lmao

nevermind. sites are back up.

July 11th, 2012 at 1:34 pmNewbie Admin Says:

GMAN:

Because bad admins bad.

Maikol you need to enter your full email address as your username, including the @yourdreamhosteddomain.com bit.

This has gotten pretty bad. Dreamhost used to be really good. Several months ago I had my database go down on the day I released one of my sites! And there have been a handful of outages since, which have effected my sites and mail. I will stick with you guys for a little bit longer, but as soon as I can I am going back to co-locating my old server at the local data center. Sorry guys, I just can’t trust that you’ll be reliable when I really need you to be.

I am sure you guys at Dreamhost are doing your best to fix this, but please let us know how long is REALLY going to take. I need to find other way to have access to my mail. I am not at my computer so I NEED webmail or to send a friendly email to all the people on my list to let them know I need to use yahoo or gmail or anything else? I don’t want to switch providers but this is really affecting me now. PLEASE give another update.

Whats happening, this is going to long, please fix it soon, is costing time and money, and i think iam not the only one

Here’s the error message I get up on the screen when I click on the re-direct link from the panel

Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

Hope that helps

Keep up the work guys.

Everyone should calm down some. The websites are still up and running. If you are using them for mail, well then yes it sorta sucks, maybe you should look at Google Apps Free and move your mail there. Keep up the good work DH, sure you will get this fixed.

Talk about bad timing. I need to add a few more things to my site before I submit it for my final Monday and I have no access. I think I’ll have to end service after this class finishes. This isn’t the cheapest web service but they have more problems than any of the other site hosting options students have used in my class.

It seems the my webserver for this domain is down too. But my other domain webservers are up.

@chris ” The websites are still up and running.”

Wrong! at least two servers have been down for 38+ hours

Squirrel Mail is down for me and giving me this error:

“Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111″

I can’t access the Panel either.

@Chris – not all websites *are* up, despite the claims in the original post. My entire VPS is down…

Hi, we are unable to reach our helpdesk website and it is impacting our customers. Please advise if this is related to the current outages or not, and investigate if separate to the known outages.

July 11th, 2012 at 1:52 pmPedro Carrillo Says:

We rely on this for all of our business…we have been unable to conduct our business for several hours…please, please, please get this worked out!!!!

Guys, if you dont like this just leave…i dont get your complaining and bitching the minute something is down. THIS IS HOSTING AND TECH, S*** HAPPENS. DH fixes it and we move on.

instead of making your stupid comments and clogging up this forum, go smoke a joint and enjoy a little downtime. Life is too short!

Cheers.

Adam – Happy DH client.

Just get it fixed!!!!

Two of our sites are still down. ebooksbudgetoptimizer.com and healthscienceebooks.com

Most of our other sites have come back online.

OK, so we were on the web panel for a client site, had–stupidly–changed everything so http://www.clientsite.com and clientsite.com redirected to http://www.clientsite.com. We realized that this created a redirect loop and went back to change the settings (so everything resolved to clientsite.com)–and your web panel went down.

Client site is inaccessible. Can you help on your end?

“healthsciencesebooks.com” that is

Hi, What happen with yours servers, too much time are them down, please fix the problems and avoid to us headache. Thank you

Luckily i only have my domain registered here, i moved from dreamhost long time ago and i come every few months for a domain name server change and bamm just when i need to change my name servers to cloudflares since im getting DDOS Attacked. they are down :(

But since i dont feel the pain of other hosting customers, ill just remain waiting and blocking ip’s manually in the meantime:P

My site is down… independentyouth.org. This has been an ongoing issue time and time again and has become a real headache.
It is crucial that this gets taken care of ASAP.

This is the second time this month that my site and email has been down. DreamHost’s twitter alternates between apologies for failed service and items that do not contribute to a productive system. Im switching to Valcato which has customer service and lower prices as DreamHost’s problems hurt my business and my clients.

My static IP website is also down. I’m not sure how many more are down. Hopefully, this is fixed soon.

Webmail down now for the third time in week. I will be forwarding all nasty emails from my customers to Dreamhost and will notify customers of my forwarding.

Robby and anyone else who needs webmail access NOW… go to http://promail.ca and use your FULL email address as username (as per my post above).

Robbie

@Jason King

You should be good now! The issues with those two sites were unrelated to the panel outage, but are currently fixed!

Dreamhost is still a good deal and has good service.

Almost 10yrs now, but im done, i was a satisfied customer for years, but email service always giving me a headache, i hope THIS TIME everything get well fixed, if not, thank you but im out.

Everyone just thank God that Brazzers.com is still up.

Wow. This is crazy. I go to log into my cpanel for the first time in almost a year, to finally make some crucial updates to my website… and find it is down. Wow. What are the odds?? (then again, I was born on a Friday the 13th)

Hourly status updates on the issue… you whiners should be thankful.

And if you are recommending Squirrel mail to your clients I don’t feel sorry for you.

@Tanya, your site is loading right now. Sorry about the inconvenience. When you’re able to submit a support ticket again, please send us the details of your ongoing issue, and we’ll be happy to take a closer look.

This is unacceptable…and problems are way too frequent. You’re costing me way too much. You should have complete redundancy…an entire ‘mirrored’ server complex.
No way to run a business…ALL of your clients are really hurting, even if you cancel all of the vitriol.
Fix it and notify me immediately.

Comments posted here may not be viewed by DreamHost staff at all.

Very encouraging…and your ‘service department’ does what exactly?

I suggest they read at least every 5th note above to get the gist…

I assume status.dreamhost.com is being hosted by Liquid Web?

what’s the email adress for support?

@Jason King: ebooksbudgetoptimizer.com is loading as well, likely the same issue your other site had. Sorry about that!
@Justin: I was able to load up http://takubanmotorsports.com/ – if you’re continuing to have issues, please send us the specifics when ticketing becomes available again.
@Robby: as per our status post, the DreamHost panel and webmail (Squirrel Mail) is currently down. Sorry about the inconvenience.
@Andy: please indicate the VPS number, so we can look into it for you.

the website idealhome.ie is down…. could be that problem didn’t solve?

@Rafael: There is no direct support email address, ticketing is done through the DreamHost panel which is currently unavailable. I was just able to load your site. Please reply back if you need further assistance.

@andrea
Thank you, the issue is the website idealhome.ie
At least is very slowly or didn’t show up

ps108069, tried to migrate from shared to it revistamultticlique.com no success and multticlique.com.br that is a revistamultticlique.com mirroed domain also getting bad http conf and sometimes 403.

Can you please check, there are 3 tickets there waiting for your help, totally unavailable and this is critical for us :-(

root@dreamhost.com:~#shutdown -f now “I do not care about my customers anymore”

Yes, you can email support@dreamhost.com but it has to be from your registered email address otherwise it’ll bounce.

On multticlique.com.br:
===============
Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”
==============

On revistamultticlique.com (ps108069)
========================================
Forbidden

You don’t have permission to access / on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
=========================================

Postcard from the team to customers

We’re in Paris, it’s great. The hotel is lovely, it has a big fountain outside and there’s a beautiful view from our window. Yesterday we went on a boat trip across the river. I loved finding out about all the different sights in the city. I tried frogs’ legs but they were horrible, they tasted like cardboard!

See you all in september when we get back !

@Robbie promail.ca is not working for me. Login failed.

@Jana Hassett: that is an error from Webmail. Please see the status updates for more information. We apologize for the inconvenience!

I have a couple sites down as well. Both WordPress installs. One on a dedicated server and one on a VPS

People are DYING. I mean not really, just need to get some work done.

Ticket #5207796 – still hoping you guys and gals get it up soon.

Hi everyone,
I checked my email using http://www.mail2web.com
It works!!
Thank you ALL for your help!!

I think we should get some free chat or callback credits in exchange for all of the problems are are experiencing. While they will not help with the current issues, it will help make DH support ‘better’ for all of us ‘email ticket only’ suckers.

Not to mention that new customers get locked into a rate almost half of what I pay now ;(

Support credits to me would help level the field in terms of all the recent problems and the new price disparity

@andrea – my VPS is homie-vserver37. It apparently had a RAID failure earlier today and was supposedly being migrated to new hardware. The message indicated that it would still be up as the engineers had mounted the backups as an interim solution.

Is this problem completely unrelated? My site has been down for over a day now. I presume your engineers have probably downed tools on the RAID repair to address this more widespread outage? I totally understand that if it’s the case, would just be nice to know when my sites will be back up…

Thanks,

Andy

If only Dreamhost actually invested in proper maintenance and network up-keep with the money that WE pay them every month, then all these outages would probably be a thing of the past…

But as usual we’ll get a “we’re sorry (with puppy-eyes) for the inconvenience”… In other words: stop whining! Who gives a sh*t about you customers!

Sad but true… Luckily there are other alternatives out there and I guess the inconvenience of moving all my domains to another provider will be worth it in the end.

All these network issues are bullsh*t!!

@andrea it occured to me you might want this number of my VPS that is down: “ps26723″

Thanks,

Andy

July 11th, 2012 at 2:40 pmMissionCriticalServers Says:

Refugee cPanel hosting on clustered Core-i5 dedicated servers / Hardware RAID / Off Site Account Backups / Special focus on security and up-to-date software / First two weeks free, no credit card, check, PayPal, etc. required, shared hosting accounts created in under 30 minutes during office hours / 24×7 ticket support / hosting@5pdx.com

Good luck with your mission. I’m sure the stress is killing you . THANK YOU for giving me time to see my family…wow there is a world out there! :-)

rest assured you have many loyal and patient clients.

# Can’t connect to 10.50.0.5 as squirrelmail
# MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

No mail today my host is gone away
The bottle stands forlon a symbol of the dawn
No mail today it seems a common sight
But people passing by don’t know the reason why

How could they know just what this message means
The end of all my hopes the end of all my spams
How could they know a palace there had been
Behind the door where my host reigned as queen
No mail today it wasn’t always so
The company was gay we turn’d night into day

But all that’s left is a place dark and lonely
A terraced house in a mean street back of town
Becomes a shrine when I think of you only
Just two up to down

No mail today it wasn’t always so
The company was gay we turn’d night into day
As music played the faster did we dance
We felt it both at once the start of our romance

How could they know just what this message means
The end of all my hopes the end of all my spams
How could they know a palace there had been
Behind the door where my host reigned as queen
No mail today my host is gone away
The bottle stands forlon a symbol of the dawn

But all that’s left is a place dark and lonely
A terraced house in a mean street back of town
Becomes a shrine when I think of you only
Just two up to down

No mail today my host is gone away
The bottle stands forlon a symbol of the dawn
No mail today it seems a common sight
But people passing by don’t know the reason why

How could they know just what this message means
The end of all my hopes the end of all my spams
How could they know a palace there had been
Behind the door where my host reigned as queen
No mail today it wasn’t always so
The company was gay we turn’d night into day

But all that’s left is a place dark and lonely
A terraced house in a mean street back of town
But all that’s left is a place dark and lonely
A terraced house in a mean street back of town

Why are all these people yammering about how great DH is? I have had more than *2 weeks* of down time in the last 6 months. If you have had problems, admit it. If not, STFU while those of us who are having issues do as DH recommended and post here and try to get something from DH about whatever DH is doing (or failing to do) to fix our problems.

Serious, stop blowing sunshine, my server is dark.

customers are chasing me, very disappointed by dreamhost down times… now the panel is not working what hack is wrong with dreamhost ? I will be loosing big business becuase of dreamhost and i was just about to order 2 dreamhost hostings next month but may be i will need to reconsider it i guess..

THE FUCK IS GOING ON

Can i get my first month of Dedicated Server paid for by you people PLEASE! :-) since this is down! hahah lulz

Hey, I really need someone to contact me by phone – it’s about my Outlook email, it’s not working.

I’m rather surprised that the MySQL server in question isn’t clustered. Yes, it’s true you need to pay for a 2nd rack-mount PC (and folks, a rack-mount PC is a LOT more expensive than that no-name clone PC you got at BestBuy) but downtime is expensive. Similarly, centralized company-wide services like the Panel web (and if separate, application) server need at least an HA configuration if not scalable parallel cluster. The MySQL cluster software is freely available, pay for support.

I wish I could get shared hosting on some nice SPARC T-series machines with Solaris 11 like I run at work but DreamHost has mostly been ok for a long time until recently.

Most of my sites seem to be up except:
http://www.fundraisingmidwest.com is down and of course my client is on a sales call and yelling at me to get is back up!!!

Help!

@ Troy “Serious, stop blowing sunshine, my server is dark.” lulz

Dreamhost in 2012:

– Major, notorious outage a few months ago.

– Email servers, spam websites, hacked sites and IP’s become all over spam lists such as spamhaus.

– June: Loads of outages. One server goes down for ten days straight, Dreamhost don’t comment publicly about it.

– July: Current issues happening right now: Webmail down, Dreamhost forum down, cPanel down, FTP not working, servers going down, at least one server ‘oranjested’ has been down for at least 49 hours (the one my site is on). The only thing Dreamhost have posted about are webmail & cpanel.

Stop trying to cover up your downtime to potential customers, and let your actual paying customers know what on earth is going on!

Moved to a smaller, cheaper and better host i guess. http://www.host4geeks.com these guys are not as big as dreamhost and I love them for that. They even sent me a personal thank you card via email :)

Site is down.

I think my WP blog has been hacked. I didn’t want to touch any of the files until DH has had a chance to look at them but the homepage of my blog has been altered (by someone other than myself) twice in about a week now.

July 11th, 2012 at 3:04 pmNathan Miller Says:

I requested this account closed 8 months ago, and I continue to get this b.s. for a service I have not used. Fix your problems so I can fix mine.

Our website is being exceptionally slow to load as well. I believe (though i can’t verify since the panel is down) that our site is being hosted on the MINELAVA server, and unfortunately i do not remember what server is hosting our databases.

Plse your help, my webmail is down!!

hornbeam.dreamhost.com seems to still be working fine.

I posted this in the forum, but I think it’s interesting:

Dreamhost have a 100% uptime guarantee, one day of free hosting for one hour of ‘unplanned’ downtime. They have stated this is unplanned, it qualifies.

Let’s say there’s 1000 sites/accounts on the oranjested server right now.

It has been down for at least 49 hours now, so that’s 49 free days of hosting to each customer.

49 x 1000 = 49,000

That’s over 134 years of free hosting they may have to give out. What’s more expensive? another mirrored rackmount server or 134 years of hosting?

Well have been using DH on and off for many years, have just renewed as well, bad move, also got an account here: http://www.wpwebhost.com. Going to try these guys for while as they claim to be dedicated WordPress guys…

I’m in the middle of moving a website. Everything is ready on the new host. I just can’t get in to Dreamhost Panel. I just needed to change nameservers. That is all.

Just wanted to pop in and say that, although this outage today is frustrating, I’m very happy with my DH service. I’ve been a customer for 6-7 years now and, yes, there’s the occasional downtime but it’s never been excessive. It’s pretty much in line with the other providers I’ve used through the years. I’ve always found DH’s customer service to be top notch though.

Panel up for me now.

Great opportunities to do host advertising recommendations, – it’s brilliant marketing strategy get the customers when there hurt. I’ve been with dream forever, and of course they have there occasional hiccups but what company doesn’t….

But don’t get me wrong I was working myself and got disconnected, unfortunate but life goes on…

Boo.. Nm error. ;(

So, is there an estimate when you’re going to be back up? I have customers waiting. This isn’t going to look good to them. I just sold them on DreamHost and this happens. Please set an expectation when you’ll be back up. Else I’m out of here.

A bunch of websites I have hosted under skybrant.com are not working. It seems to align timewise with the issues this afternoon. Can you check into this?

July 11th, 2012 at 3:17 pmBrandon Ferioli Says:

What? Downtime? DreamHost? No way!

@andrea – Thanks for coming out here and doing what you can for us. I’m sure that the downtime is killing you guys internally as much as us.

Not sure if related, but we just received a quota warning on the available space of our email addresses. AFAIK they are just redirections, so no mail should be stored on the server.

VPS also unreachable.

This is the fourth time this has happened this year. Working with ticket sales and can not wait for the server crash. You are turning this farce. Paid dearly to have a VPS and I not have a good return.

My dreamhost website doesn’t work… it’s more than just a panel issue.

DNSChanger maybe … (sorry.. had to say it)…

I need some help with installing a plugin can you help me do that?

My site is completely unavailable. Returning internal server errors. I would love to log in to the panel to let you know about this but it’s also down. This is the third or fourth time this year.

me: Hey server Im requesting my hosted page
server: http://i55.tinypic.com/8ytfdz.gif
me: ba ba ba bird bird bird, birds the word.

Moral: it sucks but yelling at them isn’t going to make it work any faster im sure they are working as fast at they can.

I was reading other folks comments… no offense, but do you REALLY think for less than $30 you’re going to get infinite disk, infinite bandwidth and infinite uptime? Really??? For those paying for the more “expensive” options.. you gotta admit, Dreamhost is still a “value” provider. I don’t expect Six sigma on the cheap. Just saying….

I am up and running with email service. I just wanted to pay my bill….good luck to all sometimes this stuff can be maddening….
but it will all look better in the morning….

@Chris – Bite me. I pay for VPS service, and the uptime is no better than the regular service.

I’d consider staying if DH could provide a single 9 of uptime. I haven’t even got that. And they sure don’t meet what they promise, let alone industry standards. I can, and have, done better on my own. If you don’t have something useful to contribute STFU.

A guy posted about promail.ca

I tried it – it said not available for dreamhost

Have I been phished?

Andy

What the… Panel is down and everything else is sluggish. Isn’t there anything in your datacenter for failover like redundant routes or perhaps a mesh topology? Someone needs to be fired.

Photo websites are also down. All I get is “One-Click Error” – “Can’t connect to the one-click database!”

July 11th, 2012 at 3:37 pmTyrone Biggums Says:

Did somebody say Crack?

Why are websites not working when they only claim it’s a web panel issue? Half of DreamHost’s sites (discussion forum, site pages, my site, etc.) are 5xx’ing. Hard to run an e-commerce site when it’s not working.

Do we all get a refund for this month’s hosting fees? Downtime stinks and there’s not much we can do about it, but I should at least get a refund.

Please note that the FILES FOREVER system is also down

@Troy If that was directed to me i was laughing because i liked the quote. I think the people that have sites down have a right to bitch. I sent in three tickets through Cpanel and never got a reply, then Cpanel went down. Still havent heard anything about whats going on with my server or when its expected to be back up. So im just as hot as you. I pay for VPS too and ive been down for almost two days now.

@Troy, I’m not sure why you chose Dreamhost in the first place since you can do better on your own? This is certainly a long outage, but most sites appear to have not been affected, only the admin panel. I would never place paying customers on squirrelmail when better options exist. If I wanted better uptime, I’d pay a lot more.

FTP and website down. Please address as this is adversely affecting our production systems.

My domain http://www.usinadoingresso.com.br already returned. But painel.usinadoingresso.com.br subdomain is still off the air. I have extreme urgency that this subdomain is working.

well it seems our website is down too

This downtime is very concerning to me.

My entire site is also down. It is more than a control panel/email issue.

Who else is just sitting around waiting for them to come back up? I have four political campaign smear sites to launch that are overdue. I might lose my job over this mess. Cripes.

Rare for DreamHost to have issues back to back like this.
Yesterday’s Webmail issue, now Complete Panel and Webmail issue.

Been using DreamHost since 2003 and they’re usually dependable,
and I’ve had Other Webhosts that had this as a common occurrence…

To those slamming DreamHost, give them a break this time.
Chances are, this probably will NOT happen again for a few more years.

Yes, this is EXTREMELY Inconvenient for *ME* @ this time, but they’ve
been quite consistently dependable for Years, so I’ll wait until they
get it done ‘cuz it’s gonna get done RIGHT…

Now, IF this was to happen again w/ in a few days, weeks, or a month,
THEN there will be some concern…

Thanks to the Techies @ DreamHost for keeping things running smoothly
UP until THIS Point… :D

Need help (if you guys are looking) with changing the redirect at a client site. We inadvertently set everything to resolve with the www, got into a redirect loop, went back to the panel to switch it back (so the URL resolved WITHOUT the www) and the panel died.
Can you fix on your end? Site is fairmountinc.com.

fairmountinc.com is redirecting to Brazzers.com now

I second Nick Ryan’s comment above. Thanks for pointing them out. Had an awesome experience with Host4Geeks.com

Nice guys, had a nice talk with their live chat agent. Recommend them! Fun guys actually.

I need to change the dns of a website hosted at dreamhost asap can someone get in contact with me?

Nothing on in the news. San Bernadino declaring bankruptcy. A rogue president. What else is there to do in the meantime? Time for Tacos!

Somebody mentioned refund of this month… I doubt they would do that, but it WOULD be nice if they gave something, like maybe a coupon for a certain amount off at renewal time?

I mean, I hate to join the chorus, but I am looking into other hosting options because of this. I only log into the cpanel about twice a year, so this means I have now experienced a 50% failure rate in being able to do so. I actually have the host4geeks.com site open in another tab right now, investigating my options…

I have enjoyed the overall uptime, hosting speed and storage space, and cost of DreamHost, but I have to be able to do my work on my clock, not somebody else’s… if it’s not resolved within the next few hours, I guess I will probably have to bail…

Thanks for your hard work. Hope you get it fixed soon!

If both firewalls are experiencing the same issue – my be is on misconfiguration. Restore from backup….

you get whatcha pay for brothers and sisters.

@Joeseph – I made the mistake of trusting someone who recommended DH several years ago. I can do better myself, but I have a day job too. I thought DH could handle simple things like network redundancy, server redundancy, and hardware restores, but they don’t seem to be able to do even those things right. It has been nothing but problems, poor service and excuses. I planned to move last year but events beyond my control made that not feasible. Looking at new providers now. Liquidweb has a good rep and is not much more expensive. I suspect I will not be on DH much past when my sites come back up.

really im hungry no one of my tickets has response from dreamhost more than an unhappy bot. some of my pages had problems more than 4days.

Dear fellow chotch IT nerds,

Keep in mind that many hosting services are just BRANDS and not individual companies. I’m curious how many other hosting services are down. Many data centers supply service to many hosts.

If you really care, have a separate service handle the DNS. Then you can switch over to a different service instantly.

___ ____ ___ ____ ___ ___ __ __ ___ _____ ______
| \ | \ / _] / || | || | | / \ / ___/| |
| \ | D ) / [_ | o || _ _ || | || ( \_ | |
| D || / | _]| || \_/ || _ || O |\__ ||_| |_|
| || \ | [_ | _ || | || | || |/ \ | | |
| || . \| || | || | || | || |\ | | |
|_____||__|\_||_____||__|__||___|___||__|__| \___/ \___| |__|

____ _____
| / ___/
| ( \_
| |\__ |
| |/ \ |
| |\ |
|____|\___|

___ _____ _____ _ ____ ____ ___
/ \ | || || | | || \ / _]
| || __|| __|| | | | | _ | / [_
| O || |_ | |_ | |___ | | | | || _]
| || _] | _] | | | | | | || [_
| || | | | | | | | | | || |
\___/ |__| |__| |_____||____||__|__||_____|

.—————-.
| .————–. |
| | __ | |
| | / \ | |
| | / /\ \ | |
| | / ____ \ | |
| | _/ / \ \_ | |
| ||____| |____|| |
| | | |
| ‘————–‘ |
‘—————-‘

Hosting on dedicated IP address (for SSL) is down. Due to issues with both the Control Panel and the regular contact form on your website it’s not possible to contact your support.

whats that supposed to be Lloyd?

Cannot FTP to sdania user. Having same problem we have with refusing password. You need to reset configuration file. You can see it has been recurring problem and the issue has been discussed on past tickets. Need it done asap, please.

Well this all would explain why I couldn’t FTP a few days ago. I think Dreamhost is kaput.

@Chris it was some ASCII art which obviously doesn’t work with this site.

we should all meet here every Wednesday night. whos in?

12 years ago DreamHost was the best! Maybe it’s time to replace the hamster wheel?

@Moon – The Hamster is Dead. Time to get a new hobby.

Im in, but im scared of meeting new people

http://i.imgur.com/GN5XY.gif

having issues with outlook as it now does not identify dreamhost as a legitimate provider…certificate issues and no ability to send e-mail…Please fix the problem!!!

Maybe Anonymous is cracking their servers again?

I’m receiving email (yesterday I got an error saying that my emails were not sent due to spam – that seems to be resolved)… but people are getting an error that I’ve not received it? Here is the error code:
melissa@getinternetmarketing.com
# #SMTP#

Original message headers:

Received: from mxout.myoutlookonline.com (mxout.myoutlookonline.com
[64.95.72.244]) by homiemail-mx22.g.dreamhost.com (Postfix) with ESMTP id
561E8200B75 for ; Wed, 11 Jul 2012 15:19:21
-0700 (PDT)
Received: from mxout.myoutlookonline.com (localhost [127.0.0.1])
by mxout.myoutlookonline.com (Postfix) with ESMTP id ABF22416F32
for ; Wed, 11 Jul 2012 18:19:20 -0400 (EDT)
X-Virus-Scanned: by SpamTitan at mail.lan
Received: from HUB024.mail.lan (unknown [10.110.2.1])
by mxout.myoutlookonline.com (Postfix) with ESMTP id 8B584416DDF
for ; Wed, 11 Jul 2012 18:19:17 -0400 (EDT)
Received: from MAILR002.mail.lan ([10.110.18.15]) by HUB024.mail.lan
([10.110.17.24]) with mapi; Wed, 11 Jul 2012 18:19:15 -0400
From: Kathy Johnson
To: Melissa Bruckler
Date: Wed, 11 Jul 2012 18:19:13 -0400
Subject: RE: Recap – Sierra Suites June 27
Thread-Topic: Recap – Sierra Suites June 27
Thread-Index: Ac1UeVc/XpxdA6YcSiihQRGzOQpoPwLOXyOwAAARwtAAAALYcA==
Message-ID:
In-Reply-To:
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/related;
boundary=”_004_F6EA88B1B9C75E4786EF844F489C2636341A716CA8MAILR002maill_”;
type=”multipart/alternative”
MIME-Version: 1.0

I say take the DreamHost team example and have your own backup.

I have webmail and that mail sits on my desktop, blackberry, playbook, did I mention DreamHost too.

hahahha hilarious animated gif

Hi, the Panel is still inaccessible! I URGENTLY need to send to an announce list. I need access to the email addresses I have there! I am way past deadline on this which was 11:30am! This is really hurting my business, and is irreprable. Please resolve this or find another way for me to send a message to my announce list asap.
Thanks.

I can’t believe it! Once again, an outage! Get moving, otherwise you’ll go bankrupt with all these unhappy customers, me included!!!

Looks like my site just went down about 5-10 minutes ago.

Sorry to add to your woes but my site is down too.

Looks like Hawkeye is inaccessible from here…

Dreamhost is folding like a California city

No mention of the fact FilesForever is also down, so customers trying to buy digital content get an error. Great for business I tell ya. So many problems. I wonder what dreamhost’s uptime is? 20%? Whatever the number, it is FAR below 100%…

I keep wondering what we are paying for here because it certainly isn’t hosting or web service!

Rough year, dudes. Still love you guys. Good luck.

This is extremely aggravating. I am a very patient person, but the amount of webmail downtown lately is getting to be ridiculous.

This is the fourth time I haven’t been able to access my email during business hours this year. And I think I’m forgetting one or two more times. It’s frustrating, and embarrassing, to have to use my free Hotmail account to conduct business because the account I pay for is down so often. Just today alone I am going to have several problems because I can’t access my email. Bottom line: I will expect a generous credit to my account when renewal notices go out.

Our site’s down; can’t reboot because can’t access panel…

You guys need to hire new support staff.

Are these people really this fucking stupid?

How does something go down for a day?

Has anyone thought of filing a class action against these incompetent turds?

My site, timr.probo.com, has had a “bad_httpd_conf” error all day.

Everyone needs to demand a free month’s service from these douches.

Hitem in the wallet. They need the fucking lesson!

well, looks like is all ok for me… for now

I’m running my MMORPG on DH… is that wrong?

I demand a free month’s for all, too D:

I got back inside the panel.

my mail just came back but my site is still down going on 49 hours

I didn’t know anything was wrong until I tried to access the Panel, just now.

Been using e-mail all day without problems (IMAP client), Google hosted mail has been fine for some of my client, and all my web services are fine (multiple domains on either a shared or VPS server).

Obviously that’s not the case for everybody here… Hey DH – would you please update this post if other servers are being affected by your network gear?

Party on the dreamhost status site!! all my sites are down. woop woop loosing money!

I only pay these guys £80 a year tho, can’t complain right?

I just got smeg on my pants

oh god there’s smeg all over my pants it’s like a scene from battle royal, Taiwanese kids 4 u

@James, I concur. For me, first databases were unreachable, then all sites were down, as well as my VPS (and the DH Panel). Now, sites and DBs are back, as well as DH Panel, but my VPS is still unreachable.

Their announcement above implies that only Panel/WebMail should be affected, and to please inform them of anything else. So, I’m posting to add that (some of?) your VPS services are also still down. Thank you for keeping us informed. The more openness the better!

full of bad aids.

@neb, yep i got cpanel back too, must be a good sign hopefull they will be back within the hour!!

My webmail is running again.

Was up for a few hours… down again for 2nd time today.

I pay $100 a year. I host a shit load of pointless sites. I can care less. I’ve been with these guys for seven years. Get over it.

Kind Regards,

The Devils Dick

And by down, I mean trying to go to any of my Dreamhost hosted websites results in “This webpage is not available”

o_O

Wow

nice try with your fake ‘DreamHost Staff’ comment. some people think they’re so witty.

Ack, databases down again…

my site is online check it out

http://artsource.tv/

yayyy free advertising

,d$b
,d$$P’
,d$$$’
,d$$$P’
,d$PB$’
,d$$$P’
d$$$P’
d$$$P’
___,,—”\ d$$P’
___,,—” \_/’\P’
\ __ ,—,_ | _/’
\ ,-” `\\ \ `)/`\
\ | \ )`-`\_-‘”‘\ \
\ `–‘`\/ `\ ) \
\ `\ `,_\-‘ \
\ \_,’ \
\ ___,,—”
\___,,—” pb

__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________
___________________________________`:#+’#,________________________________
______________________________’##########_________________________________
____________________________.#################`___________________________
___________________________+###################’__________________________
__________________________’###”’+##############+_________________________
_________________________`##+’#+”###############;________________________
_________________________#####”’+#+#’+###########________________________
________________________.##+#’+####++++”#########’_______________________
________________________#+##+#######++#++’#########_______________________
_______________________`#’#####+##+++++##+’########;______________________
_______________________:’+####”##++++++###’########______________________
_______________________#’####+’###++++##############,_____________________
_______________________#’####’#####++++##############_____________________
______________________,’####+######+++++#####++’#####`____________________
______________________#’#########,_`++++######’#+####’____________________
_____________________.##########,___:+#+######”’#####____________________
_____________________##########+_____+###+#####”#####____________________
____________________########+##______`###++####’+#####____________________
___________________';;######+#:_______’###++###’+#####.___________________
_____________________######+##________`#.+#+###’+####’#___________________
_____________________######’##.##’_____#._#++##’#####’+:__________________
____________________`######’##`___+#___;##+#+##’####+’+#`_________________
____________________`######’#+_####,___;#######’####+’###_________________
____________________’######’#`_,':+.___.##`####’#####’#:#;________________
___________________:#######’#____.`_____#+`_+##+#####’#’##________________
___________________+#######’#___________#+__`###+####+###:,_______________
___________________,#######’#___________’#`__###’########`:_______________
____________________###+###’#,__________.##,_’+##’#####’#;________________
____________________’#’####+#+___________:;#';+########’#,________________
____________________;’########_______:_.`____+++#######+#_________________
___________________`’#########_______________##++#######:_________________
__________________.’##########.______,_.____`##+++######`_________________
__________________++##########+____##+;+#.__’###+++#####,_________________
__________________+#######’####.___;,’+:#`__#####++######.________________
__________________#+######’#####`___’++#.__+##############+_______________
__________________.######’#######`________;##############.;##;;___________
___________________+##++######@###,______;###############:________________
____________________######@#########.__`’#############+####;______________
____________________########################@########:`##`###+____________
__________________,##################################;,##+_:##____________
_________________+########@######################@#########_______________
_______________`###########################@#####@#@######,_______________
______________:############################################_______________
_____________+###,,#########:###############@#;#############`_____________
____________+;;#,;#######:##_..################+#############_____________
___________.+`##:#####@#.,##:_;+###########,:##_#############;____________
__________;#.’#;########“##############;____##`+#############____________
__________#`.##+#########_+#####;###’,_______##,##############____________
__________#_:####`########,######____________##,##############____________
__________;,#####_############++.____________##,########@#####____________
___________######_;##########`______________:#,###############____________
___________.######:########’_______________;#++###############____________
____________’##############____________.;####_################____________
_____________############;’_____________,##+__`+#############’____________
_____________..######+##’_+____________.;,________:###########.___________
______________#:####_###,_:_________________________;######_###.__________
______________:.+##`’###___._________________________,#+’##+#;:___________
_____________,___######_______________________________:#_`#;._____________
_____________;___;.##;_________________________________#`_+`:_____________
____________+___,_+#;____________________________`____.#_`#_,_____________
__________`;_____:##______________________________:::’#`+.:,`_____________
_________`_______##;________________________________,`_’_';’______________
________________.:#:___________________________________._#’_______________
________________`_#+____________________________________.:________________
__________________##___________________________________`:_________________
__________________.##`_________________________________.`_________________
___________________';_.`__________________________________________________
____________________.;____________________________________________________
__________________________________________________________________________
__________________________________________________________________________
__________________________________________________________________________

“_“`_“““_`_““““__““““__““““`_““`_`
`|`|`|`|`___|`|`|`___“`\`\“““/`/__“_`__|`|`__|`|
`|`|_|`|/`_`\`|`|/`_`\“`\`\`/\`/`/`_`\|`’__|`|/`_“|
`|“_“|“__/`|`|`(_)`|“`\`V“V`/`(_)`|`|“|`|`(_|`|
`|_|`|_|\___|_|_|\___/““`\_/\_/`\___/|_|“|_|\__,_|
““““““““““““““““““““““““““`

[img]http://baregadgets.com/banner.png[/img]

CAKE CAKE CKAE

Oh, good. I’m so glad customer sites were not affected, which means that my downtime was just my imagination. (Unless I was the malicious service, which would imply that I was operating under some form of hypnosis, because I have no such recollection.)

In all seriouslessness, it does appear that my VPS is again reachable, and uptime reports over 29 days. So, the service experienced no downtime, just unreachability.

Sorry for the snarky comments. I’m glad all is well, and thanks for keeping us informed.

to all the dh fanboys: shut up

JIMBOS GOT IT GOIN’ ON YO

LOOK AT MY LADY LEGS

HAHAHAHA Jimbo is on crack. You californians are STUPID

On that subject – what is with the $150 billion dollar train to nowhere all about?

Panel seems to be back up. Still having some weirdness with webmail.

I don’t know what’s going on, but I haven’t been able to login to my wordpress site all day. I keep getting an error… The site itself is up, although it’s running a bit slow. What do I do? I have a comic to update and am stuck.

Panel is LIVE. WE. ARE. BACK. TO. WORK. OHHHHH. NO. BECAUSE. ITS. TIME. TO. GO. HOME. SEE. YOU. TOMORROW!

please touch my vaginal cord

I support hittler and I hope he wins the war.

Oh my tomatoes. My left cockle jumped in green paper knifes.

Shamp has clearly derailed

i love eating my own shit nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom

I love that Lloyd’s a bigger troll than I am.

nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnom
nomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomnomno

Please discontinue spamming of our web site. It is against the Terms of Service. Violators may have their services discontinued without refund.

Kind regards,

Jeff
DreamHost Support

How about an update on the status of homie-vserver157? 21 hours since the last caned update that says the server is up, still sucking wind here.

“Hello,
Update: All VPS and sites should be online now. The restore appears to be going smoothly at this time……”

“Customer sites were not affected.”

In which reality is that true? I’d like to move there. All our sites are still down and have been all day. And were several times over the weekend. This despite paying the extra for a dedicated server. (After we complained about service several months ago.)

i came all over my cats face wanking over your comment chris

I force lots of tiney plastic soldier up my anus because im a bipolar-sexual suffering with bipolar sexuality towards small girls and even smaller boys.

I can FTP to all my domains, files, and directories on homie-vserver157 but cannot access the MySQL databases. Is there something other than phpMyAdmin that I should be doing?

I have to fuck this woman every night

http://thinkinpictures.visualsociety.com/files/2008/04/flesh_manabu_yamanaka_72b.jpg

and the worst part is that she’s my mother.

Can you imagine how that must feel Dennis?

anyone want to see my prince albert?

you know id give you a hand nailing your mouther any time Jeff

Still having issues with my main site and WP blog not being able to resolve, saying server not sending data, blog says redirect loop, although that may be my fault as I was configuring to not use www from both right when these problems went down today….

its like 51 hours of downtime now, this is crazy. i guess ill see you all in the morning. goodnight Johnboy

Hmmm… panel appears to be working. Wonder why WebFTP isn’t taking me to my files? (just to Maildir and logs… I seem to recall being able to access more. It’s been a while, but I don’t remember going straight to this)

Our mail is still not working!!!!!!!!!!!!

Well you see the latest update….

“Update July 11, 6:40pm PDT: All issues related to this status post have been addressed so we are setting it to resolved.”

Doesn’t seem too resolved. I don’t know. Can we get an update or have they gone to bed?

dreamhoststatus.com is hosted by Linode in Absecon, NJ (someone asked that)

Shit just went down AGAIN!

FTP IS DEAD AGAIN. W T F GUYS!

All sites are down for 3rd time today.

And we’re up again… bring your toddler to work day at the datacenters?

@Bryan Harvey
Wow.. I guess its good that its accessible when dream host is offline. But I came to find out, what you say is true.

http://who-hosts.com/Dreamhoststatus.com

Not only do we not have email yet, when we logged in to the panel, we were told we had to pay the next month’s bill before they would let us reconnect. Holding data hostage for more money after a major fail. Classy.

Well, my settings are correct in both wp and panel, panel is set to www or no www and wp was never changed earlier….. So I have no idea why it is saying that the site has a redirect loop. Anyone else having this issue?

It worked for a bit earlier but now I am getting the following message again:

“Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111″

No, only issue I’m having is that I can’t actually access any of the files on my site via FTP (even though it shows up okay from the URL), and a bigger issue: the DH customer support don’t seem to have a clue. When I told them all it was showing was two dirs: logs and Maildir, their response was: “Hello,
You may need to enable the option to show hidden files in your FTP
Client. Instructions on how to do so can be found
here(http://www.rackspace.com/knowledge_center/index.php/How_can_I_show_hidden_files_in_my_FTP_Client).
I hope this helps answer your question! If there is anything else I can
help you with, please let me know! I will be glad to help!”

I responded “Should I have to display hidden files even in the WebFTP (for which there doesn’t appear an option to do so)? It didn’t used to work this way when I logged in, about 6 months ago…”

and they wrote back “Hello,

I’m sorry but WebFTP doesn’t have the option to view hidden files. It’s
meant to be a very simple FTP interface. You would have to use a client
such as Filezilla to see these.

http://filezilla-project.org/download.php?type=client

Please let us know if you have any further questions.

Chris J.”

Umm, (1) Why would you even bother having a WebFTP if your files are “invisible” and can’t be accessed through them? (2) That’s not the case; when I connected via Dreamweaver FTP and turned on viewing of hidden files, all that showed up were those two dirs and some .bash scripts.

Wonderful. Love the support. They don’t even know what their own hosting subdirs should be showing.

Hey everybody, if you find a good alternative hosting service, keep the recommendations coming (though I’ve already paid up through the end of 2012. Still, that’s no good if I can’t do anything with my site. So far in 2012 I have had a 0% success ratio of being able to update it)

My webmail is NOT working Again………….

My webmail is not working again. I have business to conduct. I’m getting very frustrated. I’m considering switching to a different hosting company.

Oh, it’s back up again… this is like a game of ping pong.

And it’s down again.

And it’s up again.

Just started working will keep everyone posted

My site is totally acting up. Images not rendering, background has disappeared. I’m use it’s not on my side. What gives?

OK site is up, evidently the changes I made weren’t propagated all day, I had to manually do it…
GL to the rest of you, and goodnight!

July 12th, 2012 at 1:43 amGreenGestalt Says:

You’ve had a few problems recently…

But, I’m OK with it:-) Been a customer for a long time and my websites were still up. Sent myself a Dustbuster from Yahoo and that got through when you resolved it so no harm really done.

Can’t say it wasn’t frustrating but big deal.

Here’s what I really want that to the best of my knowledge you provide: Freedom.

We are losing more and more and more freedom on the net to express ourselves. And my current about to launch project is based on “Anti-Political Correctness”.

But what I really want is you to keep protecting free speech, even “unpopular” speech. Anyone willing to pay for hosting, long as you can get away with it legally it’s ok to put up. A few downs now and again, no problem, happens to all the high end high dollar sites regardless…

Still no POP3 email or webmail – website does work though

So this is resolved is it?

So currently there’s no problems or outages on Dreamhost, right?

Well how about the hundreds of sites on the oranjested server that have been down for around 63 hours?

The big cover-up effort shows that Dreamhost can’t be trusted AT ALL!

My site’s been down for over 36 hours – and is still down now. So far, the support ticket seems to have been ignored (or pushed to the back of the line).

For the money we pay Dreamhost, it’s unacceptable.

Five nines people. Five nines.

What is their issue? Lightning strike? Angry employee?

Just giving quick update from the angry twitter crowd:

Oranjested down since Monday

More recently Passenger & Indians are down.

No response or public confirmation of the service outage from Dreamhost, just like their wasn’t last month when one of their servers was down for 10 days straight.

It’s a sad situation, extremely poor show from Dreamhost on a technology and support standpoint. Due to the The ‘cover-up’ efforts, this will make a for a juicy post on TechCrunch!

POP3 and webmail totally down for me – homiemail-sub 3

ugh! here we go again

July 12th, 2012 at 5:46 amClark Griswald Says:

I have been unable to FTP into my NEW domains for the past week. My old ones are fine.

site downtime has nothing to do with the central services (panel, webmail, discussion list, wiki, signup page, etc.) which is what this status post is referring to. the panel is online now so i’d suggest submitting a support ticket if you need assistance. obviously, there are a few asshats here so getting help via this page is not likely anymore.

in some cases, there are servers being failed over to new hardware. status updates for those are posted within the admin panel (not here). there is no ‘cover up’ unless you’re wearing a blindfold. of course, access to the admin panel was an issue earlier but not anymore.

“We have identified the root cause as malicious outbound activity from one of our customers that caused a subset of our load balanced firewalls to fail due to high memory utilization. Troubleshooting and resolving the issue required a number of physical reboots and configuration changes, plus adding memory to the firewalls. We’ve taken short term steps to prevent this issue from recurring, and will be implementing a permanent fix within the next 48 hours.”

i bet the same lot of you, if you haven’t put your money where your mouth is and moved to another host, will start bitching again in 48 hours when dreamhost implements a permanent fix. it’s probably one of you that had the malicious outbound activity from their account. seems to always be the loudest of the bunch who causes the most trouble for the rest of us.

Why are you telling me in the Cpanel that everything is ok?

“Hello,
Update: Aside from some central networking issues today the restore is going well. We had to restart the restoration process after those network issues were resolved. All VPS and sites should be online now. The restore appears to be going smoothly at this time.”

How come the 4 support tickets have gone unanswered?

How come no one can give me an ETA on this?

Tom, where are you getting this info from?

“We have identified the root cause as malicious outbound activity from one of our customers that caused a subset of our load balanced firewalls to fail due to high memory utilization. Troubleshooting and resolving the issue required a number of physical reboots and configuration changes, plus adding memory to the firewalls. We’ve taken short term steps to prevent this issue from recurring, and will be implementing a permanent fix within the next 48 hours.”

I dont see that in my CP or on this blog anywhere.

Email went down again on the 12th at 9 am CDST. WTF is going on. Im losing business.

Chris… blindfold? all official updates from dreamhost are at the top of this page.
Update July 11, 5:30pm PDT

I disagree that “customer sites were not affected.” :)

I had 2 sites under the same user that I was checking. One worked fine (http), the other one that used DreamHost SSL (https) did not load during the entire outage. I’m fairly certain that secure sites were impacted by the outage.

I’m glad it’s all fixed, though. Just an FYI. :)

Webmail is working… but seems abnormally slow.

There looks to be problem today. I am getting same error as yesterday. I think it just went down again.

No webmail for me. GRRRR.
Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

Squirrel mail NOT working now. Doesn’t even smell like RESOLVED yet DH. Please fix…

Resolved…. really???? In who’s reality is mail resolved, certainly not in ours?!

um, no, it doesn’t seem to be working. webmail or regular email. it say s “server not responding”…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webmail Database Issues (RESOLVED)
12:12 pm

Webmail Database Issues (RESOLVED)

Posted (July 9th, 2012 at 12:12 pm PST) by Justin@DH

We are experiencing issues with the database connection between the webmail machines and the central mysql services. Our admins are already on the case and are working on repairing this. Until this is resolved, webmail will be showing an error connecting to database and will not be functional. When we have further updates on the progress of this fix, we will update this post. Our apologies for the issues affecting your webmail service.

Update July 11, 7:43pm PDT: Webmail is up and running again since our panel (and web-based services) outage today was resolved. We sincerely appreciate your patience while we got this worked out.

Update July 11, 11:25am PDT: Sorry about the lack of updates. We were able to resolve the database issue as of yesterday evening. Unfortunately, webmail is currently down due to connection problems that we recently reported. Our teams are diligently working towards resolving this issue. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.

Update July 10, 8:54am PDT: Webmail is up and appears to be working well still this morning. Efforts to get new hardware and move the webmail databases to it are still underway. It is difficult to judge how much longer it could take but should we get a more definite idea for an ETA we will definitely let you know.

Update July 9, 8:48pm PDT: Our admins will be replacing the database with a new machine. However, due to a networking setup issue with the new replacement machine, this has been delayed. We appreciate your patience and will continue to provide you with updates as we received them.

Update July 9, 5:56pm PDT: Our admins are working on bringing up one of our central databases (which is the cause of the webmail issue). We appreciate your patience and will continue to provide you with updates as we received them.

Update July 9, 4:27pm PDT: Webmail has become unstable once again, and our admins are looking into it further now. We’ll make sure to update this post again once we have further details.

Update July 9, 3:10pm PDT The admins were able to get webmail working at this time on the original hardware, as the replacement hardware is going to take a bit longer than we would like to leave webmail offline. If we need to schedule a maintenance for moving of this service to the new hardware (much less downtime than this issue was), we will make a new status post here. For now, we will watch the original hardware for stability and re-open or update customers here should the situation change.

Update July 9, 1:51pm PDT Our admins are continuing to work on the database server and getting it setup. It is difficult to judge how far along the restore process could be but we are watching it closely and doing what we can to speed up the process. We appreciate your patience while we get this issue sorted out and are putting in all our efforts to get it done.

Update July 9, 1:12pm PDT The restore is still going and is taking longer than expected, unfortunately. It is still being worked on by our admin team and getting the full attention it deserves. We will continue our work to get this service restored for you shortly.

Update July 9, 12:34pm PDT The hardware setup, configuration and database services are completed, and now restoring the databases and last steps are being done. We hope to have this issue resolved soon, but must wait for these last steps before service can be turned back on.

Update July 9, 12:14pm PDT The specific database server for webmail is having issues that is requiring our admins to move it to new hardware. This will require hardware setup, configuration, database services setup and restoring the databases for webmail. When we have more information on the status of this webmail issue, we will update this post.

This entry was posted on Monday, July 9th, 2012 at 12:12 pm and is filed under Webmail Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

221 Responses to “Webmail Database Issues (RESOLVED)”

First comment! Come on email – don’t fail us now!

Bora logo porra, nois quer trabalhar.

I was just about to pay for 5 years of hosting. Hmmmmm……….

Man these issues are becoming more frequent…

At least there’s acknowledgment that our webmail is down…

Are e-mails sent to me going to get kicked back as well or will they still get through? I need to know pretty darn fast, could be a disaster for us.

@Travis: Emails sent via normal email clients or ftp/shell accounts on the webserver are not affected, and will deliver just fine. It is only the squirrelmail and roundcube email clients that are affected. They are used for viewing and sending email via a web interface only.

Ever heard of REDUNDANCY?

I appreciate the honesty and visibility but this downtime is inconvenient for my business especially on a Monday after a holiday week. You guys will need to learn to anticipate problems and create redundancies to prevent outages.
At some point, mere apologies just wouldn’t cut it anymore

oh phew! That’s good to know. I thought it was a total email outage. Thanks for the update!

@Mike: The http servers that host webmail are redundant and load balanced, but the database server was not yet upgraded in the many upgrades going on this year. It will be redundant in the not to distant future.

fix it please

My email is down across all my sites.

@slow: We are working on it, please see the updates above.

@Robby: Your issue sounds not related to webmail, if you are having issues with sending email via a normal email client. Please contact support and we would be happy to help.

And this is why we host email through google instead. :)

Jason – Gmail does have a good service, but they are not immune from outages either. Google gmail downtime for examples.

Thanks for being so specific in your updates! I really appreciate knowing where things are at in the process, rather than a vague “technical issues, we’re working on it.”

Thanks for keeping us updated. Although a HUGE inconvenience, in the overall picture, it could be worse. Please let us know when it’s back up and running…

Thanks for letting us know and keeping us updated – it is much appreciated as we know these things cannot be helped.

ARGH!!! My client’s email seems like it’s ALWAYS going down, and I look like an incompetent fool as a result. I’m moving her to Godaddy email, which is what I use: I’ve NEVER had an outage.

Thank you for the detailed updates, as always. Will there be any loss in emails that were sent to us during the downtime?

@Leah: You’re welcome, sorry that you are affected by the downtime. There will be no loss of emails during this downtime. Only checking email in squirrel mail and sending email in squirrel mail is not available at this time.

To all you people complaining – please just go find another web host. You’re clogging up my happy Dreamhost hosting.

A huge thanks to the Dreamhost team for being proactive and trying to resolve this.

It’s webmail, people. Plenty of other ways to get your email. Ever heard of redundant access?

I’ve been a DH customer for over 11 years. Happy customer.
I am too busy to complain, some people need to get a life or switch hosting company.
ok back to work!

The people complaining – especially the ones mentioning specific alternatives – probably work for the places they are mentioning.

Thanks for the update.

Aguardando solução.

Thanks for keeping us updated.

Bora logo que demora. Ja perdir uns $20 de quebra de contrato

Message to Ken: do not use dreamhost. The service is ALWAYS going down. Do yourself a favor and use somebody, ANYBODY else. I would but, unfortunately, my business signed up with dreamhost and I have no control over that decision.
crd

Come on guys, your status postings are starting to sound like you EXPECT these problems it’s so damned common these days!!!!!!

its working =-)

nevermind, it was and went down again.

the maintenance message is better……

Maybe the server upgrade will fix the search of old messages ……

And back down.

I do know that I’ve got more ways of accessing my email than webmail, and if I were at home where my desktop lives, I’d be using that (I really don’t want to purchase a laptop to lug around when I’ve got a lovely machine at work). Some of us have cause to want to check mail away from our home machines, and thus use webmail. Which seems to be down an awful lot.

Hi, I would love to be patient but can’t.I’m waiting for info on a job I’m booked to work on tomorrow early am and need to respond to the info .. since I can’t access anything I don’t have a number to call….This is me being impatient.. Fix the damned machine already!!!!

Come on you are killing my business without access and I have customers calling on orders I can not get to.

This is ridiculous and the last time it happened we were out for 3 days I cannot and will not put up with this my business depends on reliable email with no /minimum downtime.Gmail anybody ???

hey crd, who’s “ken”? Just a caned response you send on every outage notification?

Those of you that have desperate need to get to your email: Fire up an email client like in your iphone/ipad/android phone or install mozilla thunderbird and you’ll be getting and sending in no time.

Note: it’s not “mail” that is down, it’s webmail access to mail that is down.

Seriously LOLing at the people who think it’s a mail outage. Future Darwin-award candidates, for sure.

Time is a little different to the person standing outside the bathroom door; so please let me in.

If you REALLY need your email, you can use http://mail2web.com/ a third party email client.

oh yeah this is a major inconvenience, please fix this fast.

I need this up now. Gmail is not down, several hours is long enough. My business now suffers because of this. Please fix within the hour.

Can you add additional notification (besides rss feed) for us to receive updates to outages on this page please?
How about “Send txt to phone”

So we can’t log on and view emails but are we able to view on outlook ? Are all my emails lost or in a que ? Just left one company to join this one but maybe all are the same !!

They had major issues a couple days ago too. Couldn’t send mail. It was fixed by the end of the day. They had to reboot a mail server I think. There was no notice of an outage on dreamhoststatus which was disappointing. I’m suprised they are making this outage public.

How long?, at least tell us some information to consider our options.

Still waiting for the final solution… Sorry for the inconvinience, and best regards. Just one question: Is it possible that some information could be missed? It maight be terrible! Thanks and my best wishes for you.

I do appreciate being kept in the loop. My webmail is down, it’s a pain, but other providers have never provided me with regular updates on the situation like this.

again and again and again, brrrrrr

@Anna shillinglaw: This isn’t an email problem. Its just a problem with the software that displays your email on the web. Using Outlook, Thunderbird, Gmail, Apple Mail or other clients should work fine.

@Alonso:I think they hesitate from giving a time because if they are wrong, people will be even more furious.

After reading this page, I know now why DH support has the “Please select your general expertise in the area of this request:” selection of “Please explain everything to me very carefully”.

Seems like it might be time to add:
“Please explain everything to me very carefully, twice.”

People… just because Webmail is down, doesn’t mean you can’t access your mail. POP/IMAP is working fine. It’s like the end of the world for you people. If webmail is down… so what? Go at it another way. If your business relies to death on Webmail, I really begin to wonder just how good your business is… ;)

DH is awesome, and 99.9% up is 99.9% up, I think the “redundant access” is key here guys…

DH, keep being awesome, and I hope your admins still got take a lunch break!

The news has been filled with warning about a potential problem….this may not be related, but we should have all been prepared anyway..
Keep at it DH.

Ahhh, AWS & Google Apps, here i come…

You KNOW things have gotten serious when it has to be broken down to “the software that displays your email on the web”

SMH/FML/LOL/WTF/BBQ/ETC

Nathan
Thanks, that’s good enough for me.

If you REALLY need your email, you can use http://mail2web.com/ a third party email client.

What?,

I’m Ken

If the mail is that much important to you then use outlook instead of webmail

Word JC…

Hi Ken!

crd was warning you off dreamhost earlier. I hope you got his message.

— What?

no puedo acceder a mi cuenta de correo me indica que esta en mantencion

So, the question that should have been answered in one of your posts is, are we talking hours, days???

@Dave… you are so right!!! Come on people you really only rely on webmail for your business??? No iPhone, No Blackberry or no smartphone?

I need my hotel reservation info from my email

What?:
Ya I seen it.
I’ve been a Dreamhost customer since 2005,
I’m most likely stick around.
but when something like this happens it makes me think of going elsewhere.

The Webmail isn’t that big a deal.
Not like my website being down.

Isn’t this about the time someone says “Nightmare Host” and dh gets to block comments?

July 9th, 2012 at 3:12 pmMercury Rx Shadow Says:

It’s Mercury Rx in the shadow. This will happen at least 4 months out of every year. It’s life. Dreamhost is awesome and sometimes these things just happen, as much as we don’t like it.

I need that hotel confirmation urgently please team work on fixing the issue plz

@Dave–You’re 100% wrong. I can’t access my e-mail from anywhere right now. The issue they originally posted is a hardware issue. I can’t access anything via smartphone, outlook, etc.

WORKS!!! :)

July 9th, 2012 at 3:22 pmMercury Rx Shadow Says:

Brian Higgins, thanks for reminding me about GMAIL. That’s my backup email which is tied to my dreamhost email so if web mail goes down, I use that!!

@Travis – You must have a more serious problem with your email that you should contact support about. Because I continue to get email from my ex to my phone, I’m SURE it’s working… sadly.

@Tarik – I am checking to see if https://mail2web.com is working against DH mail servers. Hang tight. I’ll be back with an answer.

@Travis.. My POP3 is working 100%. It’s possible that you are on a different mail server than me. But if that’s the case, it’s a different problem, unrelated to this webmail issue.

https://mail2web.com does work from DH mail servers. I tried it. It seems to be an ok alternative for when something like this goes down.

My webmail portal is back up now.

@Tarik – I’m not having a lot of luck getting https://mail2web.com to connect to dreamhost’s mail servers. If I get it to work, I’ll post instructions.

Note – Once you’ve been redirected to the webmail maintenance link, reloading will only get you the maintenance page. Reload your web client page.

At least for the moment DH’s webmail servers appear to be back online.

It seems my https://mail2web.com attempts were thwarted by an unchanged password. It might work just great next time something like this happens.

If anyone tries it, I’d recommend using IMAP instead of POP3 just to make sure you don’t remove anything you don’t want to remove.

This has been happening a lot this past year. I have been a customer for many years. It is time for me to renew right now, but I have to be honest, I’m not sure I want to. This kind of thing has become a little to frequent and incredibly frustrating.

Looks like the problem just “unresolved” itself.

*poof* and it’s gone again.
at least I have IMAP on my phone still….

Status says it’s resolved on the existing hardware; however, ours is still not available — same error message as before??

Ya se les cayo el changarro de nuevo!

All emails falling down again!

Somebody tell DreamHost that it is not finish yet!

still not working
Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

webmail not working for me….

“Webmail access is currently down due to unplanned database maintenance.
We apologize for the inconvenience. Please click here for updates. “

No it’s not!

My whole website is down now, and has been for around 3-4 hours. Just after I released a big post I’d been planning for a couple of weeks.

No warning email, nothing on here or Twitter, no nothing. This is 5 days before my two year billing date. If i’d have had a warning email I could have hung back on the post.

That’s 84% uptime today, I know I won’t be offered any form of compensation or any apology like usual. That’s a 99.99% chance I’ll be leaving Dreamhost in the next 5 days.

July 9th, 2012 at 4:12 pmSamuel Lawson Says:

It’s still not working. Please fix it!

Yup… still not working… 6:09 PM C.S.T.

wtf?????????????????????????????????????????????????????????

Webmail is still down, can we please get an update??

No webmail access for four hours is annoying but these things happen. Your RESOLVED status is incorrect — I still can’t get into webmail (same MySQL Error). Your apparent inability to determine whether the problem is fixed or not is much more concerning.

Yeah, contrary to what you have posted here, webmail is still down.

Love Dreamhost and asking if u guys can please fix the email trouble we have been having all day…
working on a big international deal and really need our email up and running….
Thanks for your support,

Gordon

mine is down too :O(

Webmail still down for me too

Webmail is still down. It makes the resolved header a little annoying.

Roundcube webmail on my domain is still down. Over an hour since the issue was “resolved”.

Webmail did return for a bit. so their Resolved status WAS correct, but it has since died again.

Definitely still down…

“Webmail access is currently down due to unplanned database maintenance.
We apologize for the inconvenience. Please click here for updates.”

wow you guys need to stop freakin out. Calm down.
Go for a walk or something. Chill out. Yeaaa maaaan.

My Go Daddy sites have about 10% of the problems. My friend who is an affiliate seller LIED about their down time… switching.

the right hand really needs to tell the left hand what it’s doing. don’t send people here for updates when there aren’t any new updates and the problem is labeled “Resolved.” c’mon folks. common sense.

I was getting absolutely no message about any outages all afternoon and I am NOT a happy camper!!! :(((

July 9th, 2012 at 4:31 pmJessie Lynne Says:

Why did I just pay for hosting for a company that is CONSTANTLY having issues? come on dreamhost!

My webmail was working a short while ago but stopped while i was using it. it is currently down. fyi

Issue is not resolved, cannot access webmail directed to come here for status, and status is marked as resolved. DH you best check yourself! Issue is not resolved…

Smart idea to test new hardware on a Monday!

Mine’s just up now! seems to be working again so far!

great job, working !

Yea, it’s working fine here for now.

Getting this message:

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Whilst I have webmail access, and it works, and can receive emails on Mac Mail, I can’t send anything with Mac Mail.
Please help

My site is down
Getting this message:
Link to database cannot be established
I have an ecommerce site and in this period i receive i lot off visits
If my site don’t work i loose i lot off sales

My sites and the entire ‘Oranjestad’ server has been down for 10-11 hours now, what was your average uptime that you advertise? 99 point what? I think you need to change that.

There’s just no point in being with dreamhost anymore, why should we?

I also have no incoming or outgoing mail through any client, or on any device…been that way for 24 hours now.

Thanks for the transparency and for keeping us updated on the progress Dreamhosters! I don’t care what anybody says, y’all are all right.

Our site has been down all morning. I’m unsure when e problem began but would appreciate an update. Thank you.

Anyone else’s database just go down a few minutes ago? We’re on woodstock, and are completely broke (and didn’t previously have database issues involved with this post… supposedly).

Website down for 15 hours – now all the content won’t load. No photos loading. I feel as though someone has put me in solitary confinement and I am screaming at the walls. Anyone out there?

Please fix all the problems. It’s now been 24 hours of nail-biting frustration. Too long – especially in a customer service driven environment.

This should probably be another thread, but here’s the latest message re: the orangestad server outage:

Current Outage: DreamHost web server ‘oranjestad’ being moved to new hardware (updated 10 hours 13 mins ago)

Hello,

Update: The restore is running without any issues but is causing a higher CPU load on the server, which may result in slow site responses. We’ll make sure to monitor the progress and stability and do what we can to keep your sites up. If you experience any issues, please contact support.

As a reminder: Due to issues with the RAID controller, we have replaced the hardware for this server and will then restore all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow your hosting services to remain online for the duration of the process.

We apologize for the inconvenience caused by this unplanned maintenance. If you have any questions or concerns, please contact Support under Support — Contact Support, in your Web Panel. This message will be updated with more information as soon as we have it.

Please note, this is regarding your web server ‘oranjestad’, and no other servers or services (mysql, email) have been affected.

Thank you for your patience!
The Happy DreamHost Server Fixing Team

Thanks for keeping us posted!
Hey guys, stop comparing dreamhost against other brands. Why don’t you switch to others if you hate dreamhost and love others so much? NO ONE is perfect!

hi guys!, how long this will be?, please make it work, and don’t do this on monday cause it’s when i use more the system.

It’s been 12 hours give a up date!

“Webmail is up and appears to be working well still this morning. ”

Mine is still down. Eagerly awaiting an ETA — even a really crude one — for the rest of the task.

We use Google Apps — but our websites have been going down in 7-10 minute bursts since early this morning. Clearly this is larger than some email database issue.

My website hasn’t been functioning for a total of 20 hours now, all I want is to access the files via ftp, which isn’t working properly, and switch to a professional host. In the past 24 hours, uptime has been 17%, another four hours and it will be 0% uptime.

I’m getting increasingly nervous about this, downtime affects the Google rankings and authority we all work hard for, and since there wasn’t a post put on here, nothing tweeted to the public e.t.c. I’m worried they might try to brush this under the carpet.

By the way, if for some reason you intend to stay with Dreamhost after this, this is unplanned downtime, which qualifies everybody on the oranjested server for one day of free hosting per hour the website is down because of the ‘100% uptime guarantee’.

That’s 20 days f free hosting per customer so far, that’s a fair few years of hosting altogether if everybody claims it.

God knows who’d want to stay with a host that does this all the time, uptime guarantee means nothing if they keep going down for extended periods of time. 2012 has been terrible for dreamhost, I don’t know if I feel angry or sorry for them.

I’m with this crowd:

“Current Outage: DreamHost web server ‘oranjestad’ being moved to new hardware (updated 13 hours 25 mins ago)”

Though actually it has been 22 hours 25 minutes.

For the first 10 hours I thought I must have blown up my site myself since all anyone was talking about was email issues.

No response to a support email sent 10-plus hours ago. I imagine they wouldn’t have anything to tell me, but an answer would have been appreciated all the same.

Just adding my voice to the crowd so other oranjestad users don’t think they’re alone. Like I did those first 9-10 hours.

Unscheduled maintenance of my VPS server has rendered ALL my sites unresponsive. My support request for a fix ETA is now many hours old without response. And now I see that Dreamhost Status doesn’t even recognize this as a problem. Really? The last official update (at the top of the panel page) is over 14+ hours old. Come on DH, throw us a bone here. Some information is better than none. What is the status of the backups? Ah well… :(

Sites been down 15 hours now. No replies to support tickets, no updates, nothing. How long does it take to replace one server?

My site has been down 26 hours and counting, I contacted support and all I got is useless canned emails telling me to contact support if I have any issues.

What a complete joke. 0% uptime and no personal communication.

WHAT IS THE ETA ON THIS BEING FIXED PLEASE?

For me this has been worse than the big outage of service in January.

NIGHTMAREHOST. DREAMHOST my Ass!!! X(

34 hours and counting my website has been down. This is mega-bad.

I’m right there with you, Slice.

Even two hours longer I think. Mine exploded while I was working on it so I know exactly when it went down.

Becoming very discouraged by the lack of information.

can’t coonect !!

Is it possible to move to a new host while this one is down? Can I get the information I need from the web panel? Wednesday morning in the UK brings the third day with no email—I like dream host in principle and feel bad having to do this, but I need a more reliable service right now…

I’m trying to figure this out. Everybody moans and groans about DH’s performance these past couple of years and I know some are leaving for other, more reliable hosting. Yet when anyone in here who wants to do the same asks, nobody seems to want to share their new choice. Is there some kind of protocol against that???

Day TWO and still NOTHING. The error now reads “Site Temporarily Unavailable”. Got an auto-message from DH about 5 hours ago that read: “Update: All VPS and sites should be online now. The restore appears to be going smoothly at this time. The CPU load looks good and sites should be loading. If you do have issues with your sites or users, please let us know, so we can check into it for you. We will continue to monitor the server and keep you updated on the status of the restore process.” but NOTHING has changed. My initial support message is languishing in the queue, now over 20 hours long. I’m guessing this outage isn’t affecting everyone or this message board would be crammed with angry users. I’ve been with DH since 2004 and this is the longest downtime I’ve experienced (I think).

Im still out too and have sent a few messages to support with no reply. Why wasnt this outage reported in the status blog? Ive seen single server outages posted here many times, thats how i got to the blog in the first place, looking for info. Oh, and where do i apply for a job at dreamhost? I know practically nothing about IT but from the looks of it i dont think that matters at dreamhost. Im in it for the OT. You guys are down so much i could make a killing.

So far:

Going onto 40 hours of downtime.

“24/7″ Tech support useless since it was down, I only received canned reply.

No post on dreamhoststatus, is it a cover-up or something? YOUR CUSTOMERS NEED TO KNOW WHAT IS HAPPENING.

Tweeted about it loads, only received a ‘sorry for inconvenience’ tweet, they have not mentioned downtime at all.

I posted this on the forum in some hope of a reply with an ETA: https://discussion.dreamhost.com/thread-135510.html Join in and share your story, no reply as of yet.

Does anybody have any info? Are they trying to cover it up?

It looks like webmail is working solidly this morning – but it’s incredibly slow.

Hitting the “Delete & Prev” button (as is my habit) results in a solid 25 second delay…which makes reading a couple of hundred one-line emails a pain in the butt!

Webmail is down again… -_-

Seriously, this is becoming the new norm.

Webmail down. Server dropped connection.

My webmail is down. First thing I did was check here and I see an update for yesterday so to me that would mean it’s working. Then I went to twitter – same thing and older message. I am also unable to log in to the panel…hmmm

Major outages should get HOURLY updates on here, even if the update is only, “Restore from backup is 27% complete,” or “Moving server to new hardware, installing power supply, remaining tasks: etc. etc.”

What is wrong with the Web Panel? Will it be back up soon?

You guys have made my work life hell for the last several months. However you did teach me that a quality host is worth the cost.

After almost 48 hours without a website even the web panel that has shown me the same message that entire time is down? Is there such a thing as a storm before a calm? I can only hope.

oh dear. webmail is down.

):

Webmail down again?

Where are the updates? This is unacceptable. Looking for another host.

and down again… Difficult to do business without email access. And if I can’t do business, you’re going to lose mine.

I can’t log onto the web panel, my email is down, and it appears that the main dreamhost page is timing out. What’s up? Please update the status even if it’s just to acknowledge that something is wrong.

Well its been a couple of days… any luck? My webmail server is down for the City Government site I maintain. Have you any updates at all?

I can’t log onto the web panel, my email is down, and it appears that the main dreamhost page is timing out. What’s up? Please update the status even if it’s just to acknowledge that something is wrong.

It’s outrageous that dreamhost are shutting down my email service.
Down again, How come?
Since 2007 in this nightmare host!!!!!
NO more!!

Sorry guys, but this is pretty unacceptable.
2nd time this week there are major email problems, and I just renewed my hosting for $120 last Monday.

As stated above… http://mail2web.com works. Just use that till DH Webmail is back up. It’s just a bandaid, but it works in a pinch.

I don’t belive, again!!!

jeez get thes fricking servers up guy`1!!

Dreamhost, here is a new slogan for you guys: “We’ve been going down hill since 2010″.

Consider this the moment that I decided to give up on you guys. I am taking my 30+ domains and associated yearly domain registrations with them.

Linode here I come.

July 11th, 2012 at 1:21 pmMandy Connor Says:

Really can’t express how much I hate SquirrelMail. WHY can’t you guys keep this mail server up?? Also, why does it take 3 mins for every action I click to take effect? Click (wait 3 minutes) an email opens. Click send (wait 3 minutes) email finally sends.

Seriously? Absolute worst email server ever.

How about an ETA on when it will be back up again? Downtime is bad, but at least button up your crisis communications process. And ditto to Mandy on squirrel mail. Every time I see the stupid nuts about mail slogan it makes me cringe, worst email system ever.

me cago en le leche agria del chocho arrugado de la reputa madre los los paró !

@Mandy and others who have commented on Squirrel Mail:

The reason Squirrel Mail is so horrible and horribly slow is because whoever built it decided it would be a *smart* idea to query for all mail all the time regardless of the paging system. Ever wonder why when you clean your inbox or sent mail the system mysteriously gets quicker? There is a reason the Inbox Archiver exists — because without it the Squirrel Mail system would come to a screeching halt. The idea that a company the size of DreamHost can’t find an alternative web mail vendor to offer their clients is beyond me. I’ve been wanting that for years now.

July 11th, 2012 at 1:45 pmRabindranath Says:

Hope you can fix the problem fast.
Thank you.

U GUYS KEEP WORKING ON THIS — LOVE DREAMHOST AND KNOW YOU GUYS CAN
FIT THIS PROBLEM SOON! — WAITING PATIENTLY FOR YA’LL… THANKS.
LET’S GIT R DONE!!!

You guys know that the comments probably wont be viewed by anyone at DreamHost, right?

This is RIDICULOUS. This has happened twice with in 3 days. I need to check my e-mail and this is down grading my business by not being able to do so. Ridiculous. If this happens one more time this week I will have to move on to another provider.

This is freaking insane. We can no longer run our business this way.

You guys continue to keep working on this – I have some very important business to attend to via email from my Gmail account. My company has been suffering from all of these outages and yes pretty soon – after you sent me a bill yesterday, when my email was finally restored I wondered why Dreamhost was not compensating me for their tremendous failures, I am very much contemplating and considering shifting online services, as Dreamhost’s are getting very very unpredictable.

I am in the middle of contract negotiations and this is not good, do this stuff in the middle of the night not the middle of the day. Now I have to go to another website and send emails out saying send it here because our @#$% site is down for maintenance during the middle of the day! IT is a 24 hour business in the rest of the country! Get it figured out NOW!

July 11th, 2012 at 2:13 pmsomeone else Says:

@someone, if you’ve ever worked in a communicaitons departement you would know they certainly are.

July 11th, 2012 at 2:18 pmVincent Winter Says:

Obviously “down to make repairs” means the back-end at Dreamhost is archaic, broken and/or in serious need of investment in reliable technology. I HAVE HAD ENOUGH!!! This is silly, not professional, and a complete and utter mess. It’s time to look elsewhere for a host company.

Wow. People complaining really do not know much about the internet do they? You can get your mail from other places, via other connection portals. The public interfacing web sites are not down, just the sql database handling webmail. I’ve been a happy customer of dreamhost for nearly 4 years and I have had no complaints…. even counting the past email hiccups. I haven’t found a better hosting deal ANYWHERE, and I doubt anyone else will either. Sh**t happens, and it appears that they are dealing with it the best they can. Stupid people who think they are more important than everyone else in the world seem to be the only ones complaining. Keep up the hard work, dreamhost team. You still have a loyal customer base.

July 11th, 2012 at 2:42 pmVincent Winter Says:

Sh**t does NOT happen repeatedly over consecutive weeks, when a PROFESSIONAL company is running the show! Deal or no deal!

Change your name to DownMost

DownMost LOL!

I am only annoyed because I HAVE to use Squirrelmail right now since my email client is on a broken computer.

No doubt this error keeps happening because DreamHost’s new owners are sinking every cent they’ve got into leveraging every ISP on the web.

I need to have this up immediately. Is there anything I can do to open my email at this point?
Thanks!

for people who need email access there are a few options (sure would be nice if DownMost was telling you this…

1) you could install an email client to download your messages there – you need your main account email address + password + mail server info (mail.yourdomainname.com or org or whatever it is).

2) do a google search of “online pop email client” and choose one of those (I have no experience, so I can’t give specific advice there).

3) some smartphones have access to pop email, so download an app and use your phone.

At this point u would not like me to contact u! This is screwing with my start up! loosing $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

July 11th, 2012 at 4:17 pmJessie Lynne Says:

AGAIN? Seriously why am I spending money with Dreamhost!

This is awful. Like ‘bacardi’ I too, like many, many, many others, am losing $$$$$$ and my reputation is suffering. I’m a therapist and most of my referrals come through the internet. I pride myself on timely responses.

@ Everyone, Dam, there are other ways of getting your emails. Any one tried 3rd party software “outlook, thunderbird, smartphone ect…”

Smartphone IMAP setup:

incoming serv:
username: youremail@YOURDOMAIN.WHATEVER
password: *********
IMAP serv: mail.YOURDOMAIN.WHATEVER
port:143

outcoming serv:
smtp serv: mail.YOURDOMAIN.WHATEVER
port:587

outlook setup:

go to: account settings, new

user info:
your name: WHATEVER
email: YOURDOMAIN.WHATEVER

serv info:

account type : POP3

incoming mail serv: mail.YOURDOMAIN.WHATEVER
outgoing mail serv (SMTP): mail.YOURDOMAIN.WHATEVER

username: youremail@YOURDOMAIN.WHATEVER
password: *********

then go to more setting (button in the lower right)

pop up menu: go to advanced TAB

incoming serv: 110

outgoing serv: 587

click OK

then click NEXT, FINISH….

ALL DONE

Moon – thanks for the tip. FYI, I used mail2web to view my emails and it’s working well. Dreamhost should have posted this work around. So glad you did.

Hello Go Daddy..Good thing I switched to monthly payments and didn’t pre pay for another year…after almost 8 years as customer w Dreamhost….this will be my last..embarrassed to explain why I can’t send files to my customers and associates…sending all my work stuff through my gmail account…

I’m also seriously researching other options. These issues have come at a really bad time for me.

I’m not impressed that we haven’t seen an update in almost 5 hours. I need to access my email remotely via webmail, since I’m out of town, and this is causing me an awful lot of consternation. Having some sense of when I might have access would really reduce my stress.

Hey! It’s not just webmail that’s being affected here. Every single site I have that is using a db is down (Moonbeam). I have some seriously pissed clients on my hands here. Come on!

My entire database site is still down after 11 days …
Please fill me in and let me know if there is anything i can do to help.
This is seriously ruining my online sales and I cant check sales to pay out my team.
Heres the message i still get! (below)

Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”

We don’t give a fuck about you or your websites, go slap a brick bitch.

We got yo moniez, yo address, yo CARD DETAILS.

so SIT DOWN and SHUT THE FUCK UP

don’t make me disable this comment board shit, fags.

Aaahh, getting desperate here. How much longer? I have clients waiting on data that is sitting in an Unsubscribed Folder in my webmail. I can send and receive email through Outlook but is there anyway to subscribe to an Unsubsribed Folder without having access to webmail?

Woohoo, it is possible to pull Unsubscribed Folders into Outlook without webmail access. Easy! Just in case anyone else out there needs to do this, in Outlook, right click Inbox, select IMAP folders, and have at it!

I think its down again… Getting the same error as earlier today.

Down again!

Yep still getting the same MySQL error. Webmail is totally dead again. :-(

dead. dead. dead.

Down again. By my count, that’s at least 8 hours of no webmail access today. Which by itself exceeds 1% downtime for the entire month.

UNRESOLVED

UNRESOLVED PLEASE DO SOMETHING……………..

Unable to connect to the database!
Please contact your server-administrator.

Was connected for several hours. Now getting (as of maybe an hr or two ago):

Can’t connect to 10.50.0.5 as squirrelmail
MySQL Error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

Firefox 13.0.1

wow. it’s back. hooray?

Dreamhost SUCKS !
Problems, Errors, Problems, Errors, Problems, Errors… and the support is absolute incompetent !!!

We haven’t had connection for THREE DAYS. This is a business email account. This is a JOKE!

Prezados, Nossa conta de e-mail é comercial.
Não conseguimos entender que nos tempos atuais tais procedimentos demorem tanto!
Nos comunique a normalização.
Obrigado.

Works but is crazy crazy slow

DH = Nightmarehost !!!

Crazy slow is right, still. Webmail has timed out on me three this morning while sending messages. Very annoying.

…and now it’s broken again…and then it’s working…and then it’s super-slow…and then it’s broken again.

The blog (above) says it was all fixed yesterday – but it’s truly not. It’s just as bad as it was three days ago.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network issue in LAX datacenter (RESOLVED)
2:09 pm

Network issue in LAX datacenter (RESOLVED)

Posted (July 5th, 2012 at 2:09 pm PST) by Oscar

We are currently experiencing network connectivity issues in our LAX data center. This is resulting in site downtime for a small subset of our VPS and shared hosting customers. Our network engineers and system administration team are working to restore connectivity at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.

Update July 5, 2:28pm PDT: We experienced a brief period of degraded service resulting from  a DDoS that has been mitigated with the help of our new network hardware. Currently all services are back up and running so this matter is now resolved.

This entry was posted on Thursday, July 5th, 2012 at 2:09 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

11 Responses to “Network issue in LAX datacenter (RESOLVED)”

Thank you !!!!

Well, my site is down almost 15 hours now. I had only pain with you in the last 6 months. I have just purchased a new hosting plan with another provider and finally in a few hours I won’t be your customer anymore! You suck!

Still down.. server unreachable.. all my other domains are working fine.. what seems to be the problem? Don’t think it is fixed.

My websites have all been down for well over 24 hours now, any update on when they’ll be back up?

Not only websites are down, the mail service is down too! (you can receive mails but you can’t send)

this is a joke, i cant even work for a few hours without experiencing some kind of issue with dh. Both shared and VPS accounts are crap!

they keep sending me bills and not delivering the goods! and to all those dh fanboys who say they been here for years and not a glitch. Well you are full of it. I have been with dh for abo9ut 6years and i can say the every year they get worse. This year alone they have cost me several thousand dollars in losing jobs when trying to send prospective clients to sites hosted even on the vps account. Sites slow, Sites down, No live chat, Support replies take for ever, Constant technical hardware issues / upgrades with no to little warning and the list goes on. Dreamhost is much like apple, all fancy but not too good on delivering the goods. I just cant wait until i get time to move all my sites over to a host i have also use for nearly 2yrs with not one single glitch!!!!!!!!!! you guys are pathetic atm!!! and to all fanboys again=stop sucking up

Okay this is getting ridiculous!

MY WEBSITES HAVE BEEN DOWN FOR ALMOST 72 HOURS!!!!!!

I’m switching to Rackspace today, I’ve actually lost 2 clients over this mess!!

LAX is always slow. I thought upgrading to a dedicated server would help. No such luck.My sites were always slow or MIA, even when using cloudflare. But not anymore… I just finished moving my last site over to another host. I’m outta here!

So long suckers ;-)

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance – Non-Customer-Facing Hardware Upgrade (Orange County) Monday July 2 – 11am PDT(4 hours) (RESOLVED)
12:56 am

Network Improvement Series Maintenance – Non-Customer-Facing Hardware Upgrade (Orange County) Monday July 2 – 11am PDT(4 hours) (RESOLVED)

Posted (July 1st, 2012 at 12:56 am PST) by andrea

We are continuing with our network improvements by upgrading some more hardware in our Orange County datacenter. This upgrade should not affect customer services, this is just a notice to announce our next step in the Network Improvement Series Maintenance.

If you have any questions or concerns, please contact support with the details, so we can assist you.

Update July 2, 3:14pm PDT: The above maintenance has been successfully completed. Please keep an eye on dreamhoststatus.com, as we’ll be announcing our next steps in our Network Improvement Series Maintenance.

Update July 2, 1:14pm PDT: The maintenance is progressing as planned with no customer-facing service interruption, as expected.

If you are having any issues with your services, we would be happy to help. It’s unrelated to this maintenance, so please contact support with the details, so we can assist you.

Update July 2, 11am PDT: We are starting the maintenance within a few minutes. We will be posting periodic updates as we progress. We don’t anticipate any customer-facing issues, but please keep an eye on this post for any status changes.

This entry was posted on Sunday, July 1st, 2012 at 12:56 am and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 30 – 8pm PDT(4 hours) Part 3: Hardware Cut Over (RESOLVED)
4:05 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 30 – 8pm PDT(4 hours) Part 3: Hardware Cut Over (RESOLVED)

Posted (June 29th, 2012 at 4:05 pm PST) by andrea

Last night, we successfully completed the software-related changes (Part 2 of maintenance #4) necessary for the core router upgrade in our Orange County datacenter. On Saturday, we are performing the actual physical upgrade, Part 3.

The process will involve moving cabling of individual subsets of our network in that datacenter, one at a time. Web and MySQL service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 5 minutes). We do have several physical connections to switch so we expect to use the entire four hour window.

We appreciate your patience while we work towards completing our network upgrades.

If you’re experiencing downtime *before* the window referenced above, please contact support with the details, as it’s unrelated to this maintenance. During the maintenance, we will keep you updated, so please check back here often.

Update July 1, 12:40am PDT: We have successfully completed the hardware upgrade for our Orange County datacenter core routers. This concludes maintenance #4. We will be continuing with our network improvements, so please keep checking back for upcoming maintenance notices.

If you have any questions or concerns, please contact support with the details, so we can assist you.

Update June 30, 11:45pm PDT: We are in the final stretch of today’s maintenance. We will update this post again as soon as we’re done. We appreciate your patience while we finish up!

If you are experiencing longer downtime, it may not be related. Please contact support with the details, so we can assist you.

Update June 30, 11:01pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. Several sections have been successfully completed.
If you are experiencing longer downtime, it may not be related. Please contact support with the details, so we can assist you.
Please check back for further status updates.

Update June 30, 10:15pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. If you are experiencing longer downtime, it may not be related. Please contact support with the details, so we can assist you.

Update June 30, 9:30pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. If you are experiencing longer downtime, it may not be related. Please contact support with the details, so we can assist you.

Update June 30, 9pm PDT: The maintenance has been progressing as planned so far. We are taking a slow and steady approach to ensure minimal disruption of your web or mysql services as we go through the segments. If you are experiencing longer downtime, it may not be related to the maintenance. Of that is the case please contact support with the details, so we can assist you.

Update June 30, 8:21pm PDT: We are starting with the hardware change in our datacenter, to upgrade the core routers. Please check back here often, as we will be posting updates as we progress. As mentioned above, we expect short amounts of downtime as we change the hardware. We appreciate your patience during this process.

If you have any questions or are experiencing longer downtime, please contact support with the details, so we can assist you.

This entry was posted on Friday, June 29th, 2012 at 4:05 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 28 – 10pm PDT(5 hours) Part 2: Cut Over — RESOLVED
5:28 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 28 – 10pm PDT(5 hours) Part 2: Cut Over — RESOLVED

Posted (June 27th, 2012 at 5:28 pm PST) by andrea

Part 2: Preparation to cut over to new hardware – Thursday, June 28th – 10pm PDT – 2am PDT ( 4 hour window). With the staging completed a while back, we will now proceed to do the necessary software preparation to cut over to the new core router hardware and use it in our live network environment. Web and mysql service interruption will occur during this process, but we estimate it to be limited to mere ‘blips’ at a time. We will need the full four hour window to complete the preparation.

If you’re experiencing downtime *before* the window referenced above, please contact support with the details, as it’s unrelated to this maintenance. During the maintenance, we will keep you updated, so please check back here often.

Part 3 will be the actual physical cut over to the new core router hardware itself. The process will involve moving cabling of individual subsets of our network in that datacenter, one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 5 minutes). We do have several physical connections to switch so we expect to use the entire four hour window.

We will give advance notice for Part 3 as soon as we have the date set. We appreciate your patience and understanding while we continue improving our network!

Update June 29, 3:45am PDT: Part 2 (software) of this network maintenance is now successfully completed. We will post a new status message when we’re ready go ahead with Part 3, the actual hardware move.

If you have any issues in the meantime, please contact support with the details, so we can assist you.

Update June 29, 3:20am PDT: The maintenance is in the home stretch, still with only minimal (few seconds) service interruptions. We appreciate your patience while we finish up this phase of the network changes.
If you are experiencing longer web or mysql downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back here for further status updates.

Update June 29, 2:18am PDT: The maintenance is progressing with minimal interruptions as anticipated. We are expecting to remain within the extended window, finishing around 3am PDT.
If you are experiencing longer web or mysql downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back here for further status updates.

Update June 29, 1:32am PDT: Please note, the maintenance window has been extended until 3am. However, with the minimal (and anticipated) interruption we have observed so far, this shouldn’t really affect your services.
If you are experiencing longer web or mysql downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back here for further status updates.

Update June 29, 12:49am PDT: We are still progressing with the maintenance, with the minimal interruption we anticipated. However, it looks like it will take us longer to complete, so we are extending the maintenance window until 3am. We don’t forsee any complications, we are just taking it slower to ensure continued stability throughout the process.
If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back here for further status updates.

Update June 29, 12:01am PDT: We are still progressing with the maintenance as planned, now in the second section, with minimal interruption as expected.
If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back for further status updates.

Update June 28, 11:30pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. One section is now completely done. If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.
Please check back for further status updates.

Update June 28, 11pm PDT: We are still progressing with the maintenance as planned, with minimal interruption as expected. If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.

Update June 28, 10:38pm PDT: The maintenance has been progressing as planned so far. We are taking a slow and steady approach, and only expecting slight ‘hiccups’ in your web or mysql services as we go through the segments. If you are experiencing longer downtime, it may or may not be related. Please contact support with the details, so we can assist you.

Update June 28, 10pm PDT: We are ready to start Part 2 of this maintenance, as described above. Please check back here as updates will be posted as we proceed.

This entry was posted on Wednesday, June 27th, 2012 at 5:28 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

18 Responses to “Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 28 – 10pm PDT(5 hours) Part 2: Cut Over — RESOLVED”

Orange county is which machines or services????

How do we tell what data center or what county anything is in???????

My email forwarding was broken for more than 12 hours starting June 26 at 4:30 PM EDT. I received a notice that the mail was delayed because of issues with the “kungfu” email relay server. Then on June 27 at about 5:00 PM EDT the email forwarding stopped working again, and as of now, June 28 at 1:15 AM there have been no more forwarded emails. I have signed up with one and one internet service and I set up forwarding on my rarely used enginuitysystems.com domain, and the mail forwarding and website are working fine. I have not heard back from tech support yet. As soon as service is restored and I get my emails, I plan to transfer all of my domains. I’m afraid I might lose them if I make the change before I get the emails.

And where is June 28 a Saturday? Maybe Orange County is weird, but it’s Thursday where I live!

Everytime I try to access anything on my site it is not working, and when I go to my site it reads
Warning: require_once(/home/fashionlieswithin/fashionlieswithin.com/wp-content/themes/framework/library/functions/theme_variables.php) [function.require-once]: failed to open stream: No such file or directory in /home/fashionlieswithin/fashionlieswithin.com/wp-content/themes/eCommerce_child/functions.php on line 10

Fatal error: require_once() [function.require]: Failed opening required ‘/home/fashionlieswithin/fashionlieswithin.com/wp-content/themes/framework/library/functions/theme_variables.php’ (include_path=’.:/usr/local/lib/php:/usr/local/php5/lib/pear’) in /home/fashionlieswithin/fashionlieswithin.com/wp-content/themes/eCommerce_child/functions.php on line

HELP!!!!!!!!!!

Mails are not going. I have sent like 20 emails since morning and was wondering why no one is replying, incidentally mails are not reaching at all. Sending mail is completely down. Please help asap.!

I’m down too :/ rebooting to see if that helps …

NOTHING HELPS!

Point is I dont know how many of the mails I have sent have actually gone? I noticed the problem now, but I am guessing this is happening since yesterday.
There is no response from dreamhost.

All my dreamhost sites are down… it’s now 8:18am, PST, June 28… 6 hours AFTER the scheduled outage. What gives?

Same here, all our sites are down. It is 8:38 am PST. Is this even related to yesterday’s scheduled maintenance?

My sites are also down (Supergirl webhost).

How about you guys use your status blog and status twitter to provide some status? Novel bloody idea, I know, but some of us would like to know what’s going on and we assumed (incorrectly) that we could find out from your status pages.

it’s been over a week of our main client’s site, and all we get are automated responses from DH. We are done with this, moving all oiur customers to hostgator.com right now.

June 28th, 2012 at 10:37 amBrad in Seattle Says:

Our sites are down as well. It’s 10:30 AM Pacific.

I’m done. I am tired of managing my clients’ disappointment because of your friggin’ issues WEEK AFTER WEEK AFTER WEEK.

And there’s nothing on this page to say WHAT THE STATUS IS. Ridiculous.

You are quickly becoming the walmart of web hosting.

June 28th, 2012 at 11:16 amNick in Worcester Says:

Guys, everything is still down. Please update the status, or I need to switch hosts.

June 28th, 2012 at 11:30 amJohn Burwell Says:

How could any of these issues possibly have anything to do with the router upgrade, when the switch hasn’t even started yet?

June 28th, 2012 at 12:11 pmNick in Worcester Says:

I don’t know… but it wasn’t any of my doing.

My VPS is completely down as of 6:05PM Pacific Time. Submitted a ticket. Awaiting for response…

VPS completely down, despite my understanding VPSs were being brought back online over four hours ago. Ticket submitted # 5184703 No response to my twitter inquiry an hour ago. https://twitter.com/TrashHeap/status/218503769653116929 All I need/want is an ETA after a day of outage.

My emails were restored briefly early this morning, but after 4:30 PM EDT all emails stopped. This is the third day in a row this has happened. It’s always been around the same time. I didn’t have a chance to transfer my most important email forwarding, so I’m stuck again until they go through again. Only one response from tech support about the first problem, and they suggested that I use Google’s email servers.

This is really unacceptable and I’m glad my contract with Dreamhost ends soon. I have disabled the autorenewal.

If they knew there would be an email outage, they could have sent emails with that information. Or, maybe post a helpful status notice. I guess I’ll try another support ticket.

Please note, none of the above issues are related to the maintenance that we started at 10pm last night. If you have a support ticket pending, sorry about the delay, we will get back to you shortly if we haven’t already. If you don’t, please go to the DreamHost panel and submit a ticket under Support/Contact Support with the specific problem/domain, so we can assist you. Please also check your DreamHost panel for other maintenance notices that may be affecting your sites.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.4.1 Security Release Now Available
2:45 pm

WordPress 3.4.1 Security Release Now Available

Posted (June 27th, 2012 at 2:45 pm PST) by mir

We’re please to announce that the newest version of WordPress (3.4.1) is now available on the DreamHost panel for new installs and upgrades. Please be aware that this is a maintenance and security release and as such it is highly recommended for all customers.

You can find the release notes here:

http://wordpress.org/news/2012/06/wordpress-3-4-1/

With further details in the codex:

http://codex.wordpress.org/Version_3.4.1

Automatic upgrade (for customers that have them enabled via the DreamHost panel) will begin shortly. If you would like to see if you have automatic upgrades enabled (or want to run your own) you can find that on the panel in the same location where you installed WordPress:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

Update June 29, 4:22pm PDT The automated upgrades are rolling out and so far things look good. If you are having any issues with your WordPress installation please be sure to check and see if you were set up for an automated upgrade in case that is the issue and contact support with any questions.

This entry was posted on Wednesday, June 27th, 2012 at 2:45 pm and is filed under DreamHost Features, One-Click Installer, Security Updates. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses to “WordPress 3.4.1 Security Release Now Available”

First you guys disabled chat, now you also disabled how to send support tickets? How are we gonna contact you guys now?

One of your mail server is blocked:

521-69.163.165.183 blocked by
sbc:blacklist.mailrelay.att.net. 521 DNSRBL: Blocked for abuse. See
http://att.net/blocks

Fix your shit upgrade later assholes!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This upgrade has every blog I own down. What to do?

My wordpress sites are all down as well. What happened?

My website is down too,

What the heck is going on there now. Can somebody explain to me.

No more chat no phone number what can a customer do?

Zero customer service, all you can do is leave this poor excuse for a business and go with another company; a company that understands that their customers come first, end of story! I don’t know how much business we lost, between $10,000 & $20,000 at least. Both of our sites were down for 3 day; we missed out on hundreds of reservations.

@Steve: I cannot find your name, email or ip address anywhere in our system or in a support request. If you require any support please contact us and we would be happy to help.

@Paul: Your support request was answered at the requested time by Jose. If you have further questions please let us know.

@Ash: Please submit a support ticket with details about what issues you are having and we would be happy to look. I checked your account but can find no support requests in several years.

@Brooke: It looks like your site has already been fixed up, if you still have issues please let us know.

Anyone else having trouble with webmail today?

Yes. Webmail is timing out about half the time I try to connect

Webmail is terribel today, it times out before you can do anything.
Been a customer since 2003 but I am going to leave DH asap.

Status Update?

All done for me …

All sites done!!! Can someone post an update?

After I updated, I can not log into administration panel. It prompts me to database update, and as I update it, Error 404-page not found shows. Tried a few times for several days, it is the same thing. From outside, website works, but I can not log in inside. Thank you for your response to this matter.

all sites are now posting 403 error and no FTP access. this is getting to be regular event, sites going down with no explanation ?

After I rebooted my VPS, my site no longer come back online. What is this, seriously? This is the second time it has happened. I have already submitted a support ticket.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connection issues at one of our datacenters (RESOLVED)
8:19 pm

Connection issues at one of our datacenters (RESOLVED)

Posted (June 19th, 2012 at 8:19 pm PST) by andrea

We are currently having connection issues at one of our Los Angeles data centers, which is also affecting our DNS servers. We’re working on resolving the situation and will update this post shortly. The DreamHost panel, webmail and dreamhost.com are unavailable. We are very sorry about the inconvenience.

Update June 19, 9:14pm PDT: We are confident that this incident is now resolved; all services are up and operational. We have been checking cases in the most recent incoming support and sites are loading, but if you’re still having issues with yours, please contact support with the details, so we can assist you.

Update June 19, 8:54pm PDT: We continue to monitor to make sure all services are up and running, but we’re not calling this resolved just yet. We appreciate your patience until complete resolution.

Update June 19, 8:30pm PDT: Most of the related services are now restored, but we’re continuing to monitor the situation to ensure stability. We will update this post again shortly with more information. Thank you for your patience.

This entry was posted on Tuesday, June 19th, 2012 at 8:19 pm and is filed under General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

102 Responses to “Connection issues at one of our datacenters (RESOLVED)”

Hope this gets fixed soon D: Good luck, you guys

womp womp….and on one evening that I am actually trying to get work done. must be a sign, i’m going to go get a beer.

Wow, good thing everything is all hardened against such failures…

June 19th, 2012 at 8:21 pmCraig Mische Says:

Issues seem to be ongoing and never ending. Why should anyone stay with DreamHost?

“which is also affecting our DNS servers”

Surely each of the three DNS are in separate data centers?

This has been happening several times a month. The customer service is becoming nonexistent. It’s time to say goodbye to Dreamhost.

Dear All:

Even dreamhost.com can’t be visted, my sites are all down(www.qianfan.net), when can my sites be visted? it’s very urgent!

again…… every week!

June 19th, 2012 at 8:23 pmElise Hudson Says:

Looks like more than just webmail is affected. Maybe the mailservers too?

stuxnet or flame?

**now taking bets** :)

You must be kidding. I had a 48 hours downtime in the last week, and now my websites are all down again. What am I going to say to my customers?

ETA? wondering if I should go to bed or wait it out.. everything unreachable on Steelers. mail too.

Is there an ETA here guys? I have clients waiting right now on the phone

34 Moby Thesaurus words for “nincompoop”:
Boeotian, ass, block, clod, dimwit, dolt, donkey, dope, dullard,
dumb cluck, dumbbell, dummy, dunce, gowk, idiot, imbecile, jackass,
jerk, jobbernowl, lackwit, lamebrain, lightweight, looby, loon,
niais, ninny, ninnyhammer, nitwit, noddy, put, stupid, thickwit,
tomfool, witling

My sites are DOWN! The Panel is DOWN! The DH site is DOWN! Everything is Down! People are Dying!

Oh, here we go again, for the umpteenth bloody time this year. Dreamhost is the worst ISP I have ever experienced.

This is very bad, the worst day ever.

I have to present my website today fix it ASAP :|.

go to bed, Joe

Back up

Seems that they are back now both panel and websites. Will they remain up?

I just switched to dreamhost two weeks ago.. really wishing I had gone with someone else.

wow… dreamhost.com is down.. no wait its up..

June 19th, 2012 at 8:26 pmJohn Burwell Says:

Good luck, y’all. I, for one, understand how it goes and have no punitive intentions.

Panel back up

Again? This is just too frequent. I really have a difficult time recommending Dreamhost to anyone.

All back up for me.

I have a major DNS change that I am scheduled to make at 11:30 this evening. I need this fixed now.

many sites are down.

Panel, but not most affected servers.

Well this is the last time Im going down with Dreamhost man I was trying to sell a website BUT I GUESS I CANT now cuz dreamhost is down again … is the 2nd time it happened….. Im moving on sorry !!! Ill start looking for new hosting company.

Oh, and sites that I enabled CloudFlare on? They’re down as well. So much for your claims there.

All better now… That was fast!
Rock on support team!!

Hi!
I need to access to my app.. have important data to process!!!

dreamhost.com and panel back up, but not sites.

it seems back now, I really don’t want to see it again

I need our website to work, you guys have terrible reliability

Everything seems to be working again, it was an unplanned outage and they got it fixed pretty fast, so stop moaning!

I’m so sorry — but has this been officially ‘resolved’ and service restored to all clients? Our site remains down and we need access urgently!

many thanks

Yes, my site is still down as well. Glad others are up, but the rest of us would like our sites as well. Please don’t consider the problem solved..

Thanks for working hard dreamhost folks! To the rest of you: take a break! Have a glass of wine….

My web site is still down. Please fix asap.

kinshasa still down

Still Down here

June 19th, 2012 at 8:38 pmSaulo Benigno Says:

Still off for me: http://www.arquivoconstrutora.com.br

Thanks :)

June 19th, 2012 at 8:40 pmSaulo Benigno Says:

Back now ;)
Perfect, thanks.

I really needed you to not be down tonight, Dreamhost. It might not mean anything but it might mean everything…

June 19th, 2012 at 8:44 pmTired of Downtime Says:

Downtime? Big shocker there!
Get your act together Dreamhost, focus more on quality of service. Right now “quality” and “Dreamhost” don’t belong in the same sentence.

It is 0343 UTC and my site is still down.

We haven’t officially resolved this incident yet, continuing to monitor. If your site isn’t loading still, it may or may not be related. Please post your URL in the comment, so we can check on it for you, or contact support through the DreamHost panel, which is now accessible again. Thank you for your patience!

Please hurry, I do not have my power cord. Thanks for ruining my night.

This is a lame disclaimer btw
Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

Nothing new here. Dreamhost blows. Their service is such a joke anymore.

my site is still down

My site cloudtransfr.com is still not up.

ah!!!!!!!!!!!!!!
my sites are down again!!!

All of you should be patience. Network, servers are machines, and sometimes have technical proplems that people cannot predict.
Are you using services of any other providers? Although Dreamhost hosting is sometimes down, they always tell us what happened, while others don’t do this.
I’m using some hosting providers, but I realize that Dreamhost has the best Support Team.
Let you try some others, and feel what they do for you. Don’t stand at this mountain and look at another mountain.

the server monitoring says my server is accessible but the sites are still down.

You’re killin’ me Smalls!

Cannot access sites via http://FTP... the webpages load, but I’m trying to use FTP and it says unable to connect.

not resolved; my site is still down; submitted ticket. :(

One datacenter? NS1 and NS3 were down. NS2 is hosted off-site. Does this mean NS1 and NS3 are in the same datacenter??

Mine’s still down. Pretty sure it’s related because it was working before. http://www.showitoffdecals.com/

Thank you!

Down again for me and my site’s users :/

DH Guy 1: seen the latest think geek tshirts
DH Guy 2: yeh pretty cool
DH Guy 3: guys theres a problem with the datacenter again
DH Guy 1: cant you see we are talking! Tell someone who cares!!!

PS: If you are on a VPS try rebooting – sometimes it fixes problems…

Thank you for explaining. Been here for 8 years and have no reason to leave, thanks to this bulletin :)

@ Bill – do you have clients on your account, or is it some little website you maintain for you?

Hi, is the problem got fixed? please check my site: http://www.cmain.org, it is still down ~ 11:25pm PDT.

THANKS

Not sure if anyone is around — but our site is still down….

Thanks DH. Keep on. Fixing things can blow. Most folks understand.

you say it’s up and running but i guess i am just the lucky one who is still down. so wonderful.

Can anyone recommend a good alternate hosting company to switch to? I’m been with Dreamhost for many years, but it’s time to go; too much downtime, seriously looking for advice. thanks!

@ ERIC

There is one powerful and amazing superb hosting company but it is alittle expensive because they are using all the latest technology.. they even have SSD hard drives and many other stuff.. you can see all the hardware types and prices.. do not use dreamhost, they are using only the AMD type machines.. the powerful hosting site is:-

http://www.liquidweb.com

Only one of my sites are down…..it’s brand new…so I dunno what’s going on, but the account status is:

Your website does not appear in the Apache configuration file.

WTF???

my site & mail has been down long enough for me to miss a deadline…sweet

PLEASE FIX THIS NOW !!!!!!!!! ALL SITES ARE DOWN !!!!!!!!!!!!!!!

*sigh* Can’t do any work with this being down.

From one of my email to them, i asked them what is their hardware type and they replied:-

That is the most information we’ll release:-

AMD Opteron(tm) Processor 4122
MemTotal: 32960396 kB (32GB)

I guess this is for shared hosting or vps..

June 20th, 2012 at 11:44 amT. E. Hieatt Says:

Both my sites hosted have intermittent connectivity, sometimes loading slow, sometimes never at all. : ( Not cool.

yeah, issues again. Sites down at least 3 times in last few days. Resolved fairly quickly but tonight is a little slower.

Very intermittent at best and yeah slow as someone else noted.

My site has been down THREE of SEVEN days this week. I’m glad Dreamhost thinks that these problems are being solved. I have been able to get NO communication or ETA whatsoever. This is the worst customer experience I have EVER HAD!!!

Still a *fail* = ‘warmachine’… 4 sites in an account do not open:

“The Connection has timed out.”

Get’r done please. You’re killin’ my biz.

MY EMAIL IS NOT WORKING AGAIN. THE PROBLEM IS NOT RESOLVED!

My email doesnt work, Icant receive or send emails to anyone.

I too have not received any emails in over 90 minutes… yet I can access webmail. Sites are working. How come the problem is not posted?

We too have a client that has not received email all morning. Webmail client access works, sending works but nothing coming in and no bounce messages.

They seem to have problems with mail servers. We have been testing contact forms and do not reach their destination. And we have not done any programming changes.

WTF this is happening way to much.

sorry but, Database could no be found = errors still happening = error not resolved.

Cmon dreamhost youve been pretty good so far

zaniah restored but content is 20 days old! Everyone affected by zaniah outage, make sure your content is as it was before the outage! Weird and very bad!

There seems to still be problems relating to DNS lookups. Sites and email have been up/down all day.

WordPress sites: “Error establishing a database connection”

SMF Forums:
“Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later. ”

Now, I’m getting ‘500 Internal Server Errors’

It’s the new normal for Dreamhost. All of my sites are down. Then back up, then down; then back up. The problem is ALWAYS connection issues with the MySQL server. Ever since mid February, I’ve had nothing but MySQL connection issues on a daily basis. Dreamhost refuses to admit something major is wrong with their system.

How is Hostgator’s reliability?

Do you all mean they are outright lying?

The response to my repeated tickets was: “No other users are reporting issues.”

I have intermittent PDO/Mysql outages since June 19.

Not really resolved though is it?

I am getting zero response to requests for support. Not even the automated, “we’ll get back to you” responses.

dns mysql and ftp still down.

http working normally.

We are having financial losses.

I need this resolved soon.

Thanks.

Ok, I’m out. Goodbye Dreamhost. Good luck, guys!

The WebFTP DreamHost is down I need to fix my damn website urgently!!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.4 now available for all new One-Click installs and upgrades via the DreamHost panel!
2:38 pm

WordPress 3.4 now available for all new One-Click installs and upgrades via the DreamHost panel!

Posted (June 14th, 2012 at 2:38 pm PST) by mir

We’re excited to announce that we’ve implemented the most recent upgrade for WordPress for customers using our One-Click installs. Auto-upgrades, for customers who have them enabled, will begin shortly (so if you have that enabled you’re all set!). For those who have not selected that option you can process the upgrade from the panel where you installed WordPress:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

While this is not a security update, it is chock-full of goodness and includes “significant improvements to theme customization, custom headers, Twitter embeds, and image captions”!

The full Release Notes can be found here: http://wordpress.org/news/2012/06/green/

Troubleshooting after an upgrade?

Check the WordPress Support Forums for a list of common issues and helpful tips:

http://wordpress.org/support/topic/troubleshooting-wordpress-34-master-list

This entry was posted on Thursday, June 14th, 2012 at 2:38 pm and is filed under One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

48 Responses to “WordPress 3.4 now available for all new One-Click installs and upgrades via the DreamHost panel!”

My dedicated server is off the air again and the support does not answer. I’ve never seen a service like this so bad. Absurd!

I’d love to upgrade to WP 3.4 but since you commenced ‘maintenance’ on my VPS a week ago all of my clients sites are utterly b0rked. You haven’t even replied to my support tickets let alone fixed whatever you did to screw my server.

With such ‘service’, surely DreamHost”s days are numbered.

I love how this is posted in General Outages!!

Is that what happens when you upgrade?

Well Mike… my WP install got auto upgraded… and ever since it’s been 403 forbidden. As such it’s an outage for me… altho in all honesty the sites been down for around a week prior to this so … ;)

Will – good god, why are you still a DH customer??? If my site was down for over 24 hours I’d cancel and move elsewhere immediately.

My site is also 403. No word from support yet. Only auto responders.
I can’t access my files even over FTP and that is only reason why I didn’t move from DH yet. Client has made massive upload two days ago an I don’t have backup of those files.

What would be good alternative to DH?

I also tried to FTP my files down. No dice either.

I’m a pretty patient bloke but a week is pushing all the wrong buttons. Lucky I’m not hosting any clients myself or I’d be feeling pretty explosive about now. I really feel for anyone who is. :(

I’ve just removed two domains from DH. Two more to go.

Brief outage for my site on Soldano just now, verified with downforeveryone and Pingdom. Back up. (posting in case anyone was wondering about problems with their own site)

The entire DH network is unaccessible over IPv6. IPv4 is fine. Why roll out IPv6 if you’re not ready????

Yup… that’s the last straw. 3 VPS’s, dozens of domains. Well over $100/month, only to have A SOLID WEEK of downtime without a word from support. Just heard from them today (finally), and they claimed to fix a few issues (but didn’t). Then, they went on to say they don’t guarantee their backups.

I’m done. Moving ALL of my VPS customers to another provider. I’ll keep the dozen or so domains that are on the standard shared plan, largely because it’s a PITA to move and I’ve never had a problem with the $9.99/mo free plan — only the $100+ VPS. Ironic, huh?

Absolutely ridiculous. I’m kicking myself for not having moved everything after the last outage.

/rant.

I would move my domains in a heartbeat but worry about other providers not being much better. Any of you have a tried and true reliable provider to recommend? I sooo gotta make this move. There is no other level after pissed off…

all my sites all down about 12 hrs,why?Sever is pulcherrima

same problem here, my sites under pulcherrima all day down =(

@Andrés

nobody answer our Tickets.why?the down time was over 13 hrs

Your *entire* network is down.

Hey!!! you are out right now

sites and PANEL down

My websites and mail are all down.

everything is broke no panel no mail no sftp nada

They have terrible response times from when their stuff goes down to when they at LEAST say “We know there is an issue” on here or twitter.

My sites are all down, no cpanel.

June 19th, 2012 at 8:12 pmFUUUUUUUUUUUUUUUUUK Says:

down

Gary: Yep. If they can’t even get a CS rep to post “hey, we’re aware of the issue” I shudder to imagine how slow their network team works.

Have spoken to DH, they confirm knowledge of an issue at the moment relating to one of the data centers. Some of our sites but not all are down. Those we have on VPS and those on DS are fine in their datacenters (thankfully).

Response from DH:

“This was an unplanned outage, which appears to be coming from the datacenter connection itself. If all goes well, should just be a few minutes

However, I don’t have a more accurate answer or ETA available at this time unfortunately”

All of my sites are down. Can’t even get to the dreamhost site. SO frustrating!!

AHHHHHHHHH Everything’s ruined.

here’s some more dead bodies:
panel.dreamhost.com is taking too long to respond.
blog.dreamhost.com is taking too long to respond.
wiki.dreamhost.com is taking too long to respond.
discussion.dreamhost.com is taking too long to respond.
somebody took the blue pill and flipped the wrong switch

All sites down again, apache configuration error this morning. So what’s the cause now?

Jay,
That is nice, but they should be immediately posting on twitter that they know there is an issue. Even if it will be for only 5min (though it has been at least 15+ now.

Why I persist I don’t even.

Panel is down, dreamhost.com is down, there’s absolutely no way to contact Dreamhost. Everything is down that I can find :(

Someone on the crack networking squad tripped over an ethernet cable again did they?

Anyone have a way to contact ANYONE at dreamhost? Even an ETA would be nice

Lebron is injured!

Move hosts to where dreamhoststatus is hosted?

evrything back up .. that’s right take the red pill

this is absolutely hilarious…. how much money do they make?? they must be spending it all on junk food , ergonomic chairs and think geek gadgets…. And all those imbeciles talking DH up, guys i have two other hosts that are just as much as my VPS account and have had no more than 1hr of downtime for a whole year!!!

I’m moving my small clients when i get time. They were good back in the day but now its just one issue after another.

jimbo, who are your other hosts? I’ve got my credit card out.

WTF is going on?? All sites down again!

it’s so annoying

dreamhost what is happening !!!!!!!!!! again all sites are down !!!!!!!!! cant even connect to the website !!!!!!! just throw away all your stuff and replace with a new high tech machines which uses SSD and all the latest technology !!! almost every single day your server are down !!!!! come on man, this is not the way you do business and CHEAT the people !!!!!!!!!!!!!

Been down for more than an hour now… can’t access our http://ftp... sites not loading.
Please help?!

Thanks

Several of us keep asking who are the other more reliable hosts that many here keep blabbing about moving to, but nobody seems to want to share the info. Yet, moving is DEFINITELY WARRANTED. What gives? You guys keeping your discoveries close hold for some reason????

Can anybody say if hostgator is any better, more stable/reliable?

Site DOWN, again!!! :((((((( I’m sick of this SHIT! Fuck off dreamhost!

webmail down

what’s the story with my wordpress site getting hacked by someone calling themselves ‘brkod?’ i have a site that got hacked and says as much in the site title. and even after i deleted the hosting for that domain, it remains live. when i try to delete it via FTP i get a ‘permission denied’ error.

still no response from DH on the support line.

anyone else out there have this happen to them?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » DNS Server Problems (RESOLVED)
5:15 pm

DNS Server Problems (RESOLVED)

Posted (June 13th, 2012 at 5:15 pm PST) by Ralph

We are experiencing some service interruption as a result of an issue with our DNS servers. This will cause sites to load slowly or hosting and email may not be working. We are working quickly to resolve this and will provide an update as soon as possible.

Update June 13, 7:55pm PDT: After waiting a bit for all issues to subside, we can confidently say this problem should now be resolved. All DNS servers involved are behaving normally, and are responding correctly to all tests. We have spot checked many sites, as well, and all perform normally. If you’re still seeing problems, please let our support staff know immediately! — Brian H

Update June 13, 6:45pm PDT: We have found some causes and eliminated them, but we are still investigating the root cause of the issue and working on getting your services up and stable again. All three of our name servers are working, but we are continuing to monitoring stability. We are confident that we’re very close to getting this incident resolved. Service may still be slow until then, so we appreciate your continue patience.

We apologize for the inconvenience this is causing. Please note, your web and mail servers are up, so as soon as we get DNS working correctly, your services will be immediately back up.
We will post updates as we progress towards resolution.

Update June 13, 6:14pm PDT: We are still investigating the root cause of the issue and working on getting your services up again. All three of our name servers are working, but we are monitoring stability and we won’t quite call this resolved. Service will remain slow or unavailable until we stabilize all three of them. We apologize for the inconvenience this causing and thank you for your patience. Please note, your web and mail servers are up, so as soon as we get DNS working correctly, your services will be immediately back up.
We will post updates as we progress towards resolution.

Update June 13, 5:40pm PDT: We are still investigating the root cause of the issue. Two of our name servers are now responding correctly and we are still working on the third. Service will be slow or may even be down if you hit the problem name server. Our teams are working together on this to resolve it. The servers are up, so once we get DNS working correctly you will be immediately back up.

This entry was posted on Wednesday, June 13th, 2012 at 5:15 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

80 Responses to “DNS Server Problems (RESOLVED)”

Yup, all of my stuff is down.

My site is down too

Yes – I cannot access anything and I have a huge project due tonight. Please hurry!!

Yep mine up and down randomly too. Argh. Trying to edit my site ATM and it keeps going offline.. Super annoying.

What happened to live chat? it has been non existent for a long time, coming here has become a habit for me. I sit at my desk every day for at least 10hrs a day and i can honestly say that both my shared and VPS account has had more downtime and problems in the past year than i would care to mention. I dont understand how a company that posts on facebook their funky photos of their building and shows off their success can on the other hand continually provide a very second rate service even on VPS? not happy DH – when i get time i will be packing up and moving on to a very reliable host that i have some clients on now and not once in a whole YEAR has there been a downtime issue and yet it only costs a few bucks more than the VPS with you guys! Whoever runs your company needs to have a good look at spending our money on some better hardware / software and maybe even staff!

One of my VPS is responding, all sites on my other VPS are up. Hmm.

Support Team– we love you! please bring our servers back!!!

I reported this problem to DH at 2012-06-13 16:10:17 … and this post has been started after over 1 hour at 17:15 …
Not sure this is just dns since my errors on control panel reported that all my user folders were all missing/gone …

Yup. I’ve been having related problems since yesterday night. Glad they verified at 5:15 Pacific. Should have played golf today…

.. sigh

Confirmed this had been an attack. Cloudflare homepage also affected, getting 502’s

My site has been up and down all day, certainly before 5pm pacific. Hope this gets cleared up, and a little more information wouldn’t hurt either.

Go dreamhost techs go!

Diego! Did not see your post/we were posting about the same time. Same here re: applications installed. I have one click WP. Checking via FTP all the files are still there, but DH panel/manage apps says no applications installed …

Well I am back up now, I hope this can be figured out. Just one question. We are supposed to get 99.99% uptime right? Just wondering because I have already experienced 2 downtime issues in my first month with dreamhost.

dead in the water

FindFresh, I have had 2 downtime issues in 4 years with DreamHost, and the second was today. I think that’s a darn good track record. Sorry to hear of your issues, but the response time for large and small problems has been terrific and I am impressed with the level of service overall. I hope you become as much of a fan as I am. :-)

Email works but nothing else.

I just got a reply from tech support that my site works fine for them, and maybe my I am having a problem because my password is incorrect. W…T…F…

I guess I’m going to have to completely break away from Dreamhost because you folks can’t even get DNS right. I moved hosting of my business site to Linode–highly recommended–thinking that I wouldn’t have connectivity problems that have plagued my site over the past two years. It looks like I’m going to have to completely cut Dreamhost off and change my DNS servers, too! Dreamhost is the most incompetent web hosting company on the planet–member since Nov. 2008.

The perception of your datacenters, in my mind, is rows of old dusty computers with duct tape holding them together. There’s absolutely no coherency to the place, it seems.

@lance

Finally some one who is making sense and not kissing butts or trying to sound all super right and politically correct LOL

I am moving soon too, im over this after this many years on shared and VPS i can count several times on both fingers and toes the amount of downtimes and unschedueled tech issues.

There are hosts out there that work amazingly well. I have one now that has still not experienced a hiccup in one year! their status list is virtually dead apart from uptime all good reports!

DH is really like WP and APPLE all shiny and promoted well but not much of a goods rep with service. I say this about WP because it is the biggest contributor to sites getting hacked on peoples shared servers. Because so many web guys use wp plugins and dont even know whats in them!

anyway theres my second rant

my sites are still slow, not fixed.

were seeing the same as Jim sites are still glacial

this can’t have been fixed. my site just isn’t loading.

Fucking losers. The bottom line is that the issue is trust. And that trust is dead. No confidence, no reporting, and no solutions.
Please die.
Walmart is hiring.
Douchebags.

My site’s still extremely slow, several minutes to load.

Is DH’s business plan something at the level of working off the seat of their pants or what?? It seems as if it was born in someone’s garage and never left that mentality when it grew out of the garage. Nothing but reactive operations with no long term plan, and that’s why they get bitten by something (attack one day, infrastructure weakness another, equipment dying the next since no tech refresh plan)…Geez DH, the boy scouts can provide you with an adequate business model to follow.

Sites still down….

Still getting timeout and delays. NOT FIXED. Don’t be so quick to declare things RESOLVED until you actually test.

can’t connect to database server

NOT RESOLVED! Still experiencing intermit connectivity…though the duration seems a bit shorter.

Problem solved as of this morning…updated a plugin…started to work on another project. BOOM..database error. I am getting tired of this $#%^

The problem is not resolved!! My VPS still getting intermit connectivity and all my clients yelling in my ears!

ps34010

My site is STILL loading VERY slow. And since three days now….I am astonished, I’m losing visitors and money!!!

our sit is not loading and our email will not go out. It says we’re over our limit even though we haven’t sent any out recently.

Thanks

I came back online this morning to review where things stood. Glad to hear you sorted out the main issues.

I have to wonder, however, if the down wiki is part of the same issue? I’ve been seeing this occurrence frequently the last couple months. Sites are down, then back up, and when they’re down the support blog and wiki are running, and a couple of times (today one of them) something is wrong on your domain.

Today, the wiki is down.

My site is down. MYSQL is down. I am not getting reply verification by email of either support ticket. This place if a f@#$ing mess. I have wasted the past 2 days of my life on Dreamhost. I have experienced more down time than up time. I would be a fool if I stayed with this service. It is going to be a hassle and another 2 days of my life to change all of my DNS again…wait 24 hours…load WP and all my plugins elsewhere, but it looks like I am going to have to do this. I WILL NOT BE SILENT ON SOCIAL NETWORKS ABOUT HOW BADLY THIS SERVICE SUCKS.

Nothing…I am getting no replies, no email confirmation of support tickets. Emails to support are bouncing. Dreamhost is dead.

One of my sites is still down this morning.

Down again this morning?

Hi,
Please fix yours DNS issue ASAP! We still have issue with DNS some ours web site don’t work!!!

My website was fine yesterday, but it is down today.

My VPS was up last night, looks down this morning. Contacting DH support.

Once again, a DNS issue has made my website dead in the water. 4th time this year. I normally don’t rage about uptime, but something significant is wrong. Not only do Account Tests no longer work, but Live Chat no longer works. (And I pay for it.) Sick of it. You’re losing an almost 5 year customer on multiple sites, not out of anger, but out of necessity. I can’t pay for a service that doesn’t care to tell me the truth, makes me find out for myself that sites are down, etc.

Okay… it’s 11:03AM here in CT time zone… My sites are all down. They WERE up a few mins ago but now they’re down. Please fix this. This is becoming an all too common occurrence I’m starting to feel worried about the safety of my work….

Looks like it just went back up!

Been here since 2006. It has been this way since 2007. I need to shut my account since I only use DH for email now and they can’t even get that right.

What’s really pissing me off, is the absolute lack of support. I can assume that in a small business, the same guy that fixes the bugs, talks to the customer.
But in this case, it’s clear. They dont’ know what the fuck is going on, and are trying to avoid revealing it to their users.

No response to tech support requests, and the same fucking bullshit in their “Progress report” .

Fix the fucking thing or get the fuck out of the business.

End of story, I’m going elsewhere.

I have 233 domains on this turd bomb. CYA loser!

For the past few days the only way to get a hold of support is via callback.

All week I have had support tickets open for 6-12 hours with no reply. Or if I did get a reply it wouldn’t be helpful and I would have to wait another 6 hours for another reply.

However I can submit a callback and then get someone to fix my problem within an hour. Never fails! So thank god for callbacks.

So basically Dreamhost seems to manufacturing outages to generate callback revenue?! or they are so swamped with issues they are only dealing with callbacks?! Either way it is scary!!

Because their email support is unusably slow these days and live chat is completely offline.

I remember the days when you could go onto live chat … get an issue resolved in literally 10 minutes. Now it’s 10 hours if you’re lucky.

I’ve been getting random MySQL connection timeouts for the past two days so the DNS issue is not resolved completely. Albeit improved.

Keep on fixin Dreamhost.

Sites been down since 1AM Monday. Fix the damn thing.

Hi there,

I still have several DNS’s not working (this is after your Update June 13, 7:55pm PDT, where you say everything is working). What to do?

Thanks

Update: This was my -awesome- response from Dreamhost. Apparently, I’m all to blame for the server downtime. Just thought I’d share. My blog is a very simple wordpress site. Not many plugins. It’s a homebase for people to find me. Nothing more.

I am sorry but I was unable to reach you this morning over the phone.
Because of that I will not charge you for the callback.

I’m sorry that you are encountering these issues. It would appear that
the sites that are running under your user “girlmeetsgeek” are
occasionally hitting our shared server’s memory limits. This is why your
site is seemingly slow, and the 500 errors are directly related as well.
Here is an excerpt from the log kept by our Process Watcher, the daemon
that is killing your troublesome php5.cgi processes:

Thu Jun 14 09:05:49 2012 procwatch3 INFO: PID 59303 (php5.cgi)
girlmeetsgeek:pg1571228 : killed for uid ram
Thu Jun 14 09:05:49 2012 procwatch3 INFO: PID 59308 (php5.cgi)
girlmeetsgeek:pg1571228 : killed for uid ram
Thu Jun 14 09:05:59 2012 procwatch3 INFO: PID 59348 (php5.cgi)
girlmeetsgeek:pg1571228 : killed for uid ram

Procwatch is a daemon that runs constantly on shared servers to monitor
the usage of RAM/CPU and execution time so that no single user can use an
inappropriately high percentage of the shared resources and impact the
overall health of the server or the server’s ability to serve all users’
pages.

Now, it’s important to get these errors fixed to keep your sites running
with optimum speed, as well as keep our servers happy. Unfortunately it
can often be a bit of a trial-and-error procedure, but I will provide as
much information as I can to aide you in this process.

Firstly, it should be noted that any sites running under your user can
contribute to the problems you are encountering. For example, if site A
is running processes that are taking up 85% of your individual limit, and
site B tries to run a concurrent process that takes up 20% of the memory
cap, site B will have it’s process killed since it is the one that took
you above your limits, even though site A’s script is the obvious memory
hog… For this reason, it’s important to keep all of your user’s sites
optimized and running smoothly.

There are quite a few things that can contribute to high memory
consumption on any given site, but I will mention the most common
causes…

Plugins – Especially third-party plugins can often be poorly written and
run up the memory consumption. I often recommend disabling all
non-critical plugins and see if the problem gets better, then enabling
them one-by-one until you are able to identify one that causes problems.

Database related overhead, or an unnecessary amount of database queries –
If you are trying to query your database for thousands of results at a
time, this can also cause issues… Try to keep all database sizes and
queries to a minimum by deleting irrelevant or old information from your
database. You might also try optimizing tables in your databases by
visiting your database through PHPMyAdmin. Simply use the “Check tables
having overhead” link located directly underneath your tables and then
“With Selected:”, choose “Optimize Table”. This might reduce overhead in
your database, which is basically a lot of empty and redundant space that
can slow your queries down.

It is also possible that a spike in traffic, or a heavy run by a site
indexer (such as the GoogleBot), or even abusive hits by a given IP
address, could cause this on a short term basis, so you might want to
inspect your access logs for such activity (you can often manage this
problem by implementing a robots.txt or by blocking abusive IP addresses
with .htaccess).

You might as well want to try looking over the suggestions offered at our
wiki pages on this matter:

http://wiki.dreamhost.com/Poor_performance
http://wiki.dreamhost.com/Optimize_database
http://wiki.dreamhost.com/WordPress_Optimization
http://wiki.dreamhost.com/Database_Maintenance
http://wiki.dreamhost.com/Slow_site_troubleshooting
http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage

Implementing the above suggestions more often than not will fix the
ProcWatch errors, but if you are still finding that your sites encounter
the same problems, you might want to consider moving to a VPS
(http://wiki.dreamhost.com/DreamHost_PS) or dedicated server where you
can reserve sufficient RAM for your own processes to run!

I am sorry if this news is discouraging, but I felt it was important that
you have the full benefit of knowing what is occurring here so you can
make the appropriate decision to meet your needs for this site.

If you have any other questions, or if I can help you further, please let
me know! I hope you have a great day!

Thanks!
Matt S


DreamHost Support Team + support@dreamhost.com
Earn over $97 for each referral: http://www.dreamhost.com/affiliates/
To continue this support case, just reply to this email.
Open a new case at: https://panel.dreamhost.com/?tab=support

We are still a problem.

Oh look! My site has had ONE page view today. Hmmm could it be that no one can GET TO IT?

All my sites on the ‘spawn’ server have been running slow or not connecting for the last 7 days! It can take over a minute for the sites to connect and start download. All I am getting from Dreamhost support on their daily reply to my support requests is ‘it must be my fault!’. Considering that there had been no changes to my sites during last week, and they had been working successfully for the last few years, I find that reply without foundation. For 24 hours things returned to normal over the weekend before collapsing again. Connection problems are usually DNS based but Dreamhost support have denied there was a DNS problem for 7 days!

I have been a customer of Dreamhost since 1999 but after this latest support debacle which has still not been resolved I will now start looking for another host I think.

Both of my sites are down. They were working about a half hour ago, and now nothing. It seems pretty obvious that everything is NOT fixed.

Still does not work. I wrote support 4 hours ago, but no answer.

Dreamhost is a cheap hosting company (I like), with many features (I like), but extremly unstable (completely unacceptable for business use). I don’t know what to do other that to start looking elsewhere – although I did not otherwise want to :-(

I had to put a ticket in, our DNS is NOT being resolved, its like we do not own the name!

none are available yet Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@330atlanta.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Website loads, email still down. Looks like all of Dreamhost’s pages are down too :(

Anyone still having the site loading VERY SLOW? The problem is NOT RESOLVED for me…

Not sure if it’s related to the DNS issues or not, but for the last 24hrs or so my site has been unable to connect to Google servers (I use the Google API to pull photos off Picasa for display on our site).

Support ticket raised ~14 hours ago – no response at all yet.

I have the same problem of Richard, i use Google Recaptcha and doesn’t work from 24+hours, i have sent 2 mail to support, no response, i can’t use my site…..

My site simply hosts a gallery2 setup, and it’s been about 20 hours since we’ve seen it..

All of my clients’ sites down for almost a week now. No human reply to my support tickets. I pay $100/month for a VPS. DreamHost’s service is appalling, completely unacceptable. I will be moving elsewhere at the earliest opportunity.

DH customer since 2008.

This issue is certainly NOT RESOLVED. Please remove this misleading information from the title of this post. All my sites have been down a week. No human has bothered to respond to the support tickets. Take responsibility for your actions and honor your customers.

My sites are still not working :-(

No update from support despite my requests :-(

It’s been like this for more than 24 hours :-(

Please don’t claim that the problems are RESOLVED – its not true, and you loose all credibility that way :-(

I’m so sick and tired of all these outages… :( Owww heyy… I just got my account auto-renewed this week and the service still sucks! Does it mean I have to endure for another year? (Crying out loud)

Sigh…

I am having a lot of trouble with my site today. The pages are not fully loaded or just hang, making update. I’m not able to make updates via WordPress.

Site has been down since 1AM Sunday. I just renewed two weeks ago but will be leaving also.

One week after reporting my connection problems with server ‘spawn’, and multi-daily emails to support, it has finally been resolved …… I insisted Dreamhost move my sites to another server. They did this yesterday and ‘magically’ the 1 minute to connect and download a page times have dropped to their pre-problem 1.6 seconds….. hoorah I can hopefully stop losing money!

Well guys my site appears to have been HACKED – some kind of parked page at the moment. It’s been like this for about 24 hours now. This is my company website and not only do we look like we haven’t paid our bills (we have, obviously), our email doesn’t work either. Four urgent “omg everything is broken” tickets to support and not even an acknowledgement yet that they’ve received anything and are working on it.

Dreamhost support used to be great. Have their staff just stopped coming to work or something? If it’s not sorted by Monday I’m moving all our domains elsewhere.

All 5 of my websites have been down for A WEEK!! I put in a ticket every day and just get a robot letter response. THIS HAS NOT BE RESOLVED!! Please fix!!! I am losing business!!

My web site is down…this happens far too often these days.

My site had also been very very very very slow of over a week. The load on my shared sever is only 2.0 right now! This status page is a facade of transparency! What is really going on over there?

+1 to Herman’s comments. Being slow was atleast something we could live with. But now, all my sites hosted on DH are down. This is ridiculous because i am running a media portal and cant afford to have downtime! :(

My site is still down.

I am new to Dreamhost. I tried to map my WordPress site to my domain name. It’s been about 12 hours since I’ve done all the DNS mapping and still nothing. Can anyone tell me whether that is normal, or due to the DNS problems Dreamhost is experiencing? I sent Support an email yesterday, but have yet to hear back from them.

@Nijish: Your site is loading and the mysql server issue has been resolved.

@Herman: Your ticket has been answered, the apache issue has been resolved.

@Nevena: I don’t see any recent tickets from you, please contact support via a ticket to receive faster attention to your issue. I believe I have fixed some of the slowness, but please make sure your wordpress sites all have wp-super-cache.

@Rox: Your site has now been fixed.

@W. Brown: DNS changes normally take 4 hours to complete. I see no ticket open from you, please submit a ticket from the control panel if you need further assistance.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 9 –9pm PDT(4 hours) Part 2: Cut Over(REVERTED)
8:59 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 9 –9pm PDT(4 hours) Part 2: Cut Over(REVERTED)

Posted (June 9th, 2012 at 8:59 pm PST) by Ralph

Part 2: Cut over to new hardware – Saturday, June 9th – 9pm PDT – 1am PDT ( 4 hour window). With the preparation and staging completed Thursday night, we will be able to cut over to the new hardware and use it in our live network environment. The process will involve moving individual subsets of our network one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 15 minutes). We do have multiple physical connections to switch so we expect to use the entire 4 hour window.

Update June 10th, 12:50am PDT: We have successfully reverted our changes and all sites affected by this change are working correctly again. We were able to work through some of the challenges that came up during the maintenance, but at this point we feel it is best to take the information gained and re-evaluate the best way to upgrade the core router at our data centers. We will provide you further information once we reschedule part 2 of Network Improvement Series #4.

Update June 10th, 12:25am PDT: Services are restoring and most all test sites are up. We are also testing sites submitted in support requests and in the status post comments as well. Thank you for your patience and understanding during this maintenance window.

Update June 10th, 12:00am PDT: We are reverting back to the previous state. We were unable to completely iron out some of the issues we were seeing after the switch of the first rack of machines to the new router.

Update June 9th, 11:00pm PDT: First row is still underway. We are seeing some service interruption and we are working to get your sites up. We are currently rebooting the switch and expect to be recovered once that is complete.

Update June 9th, 10:00pm PDT: We are starting our first row of machines now. People with machines on this row will experience an inability to connect to their site or their database for less than 15 minutes.

Update June 9th, 9:30pm PDT: The final staging work that we needed to finish today is now complete. We are near ready to start switching over individual networks. Once we start actually moving the networks we will let you know.

Update June 9th, 9:00pm PDT: Our team is here and ready to start on the maintenance. We will keep you informed each step of the way.

This entry was posted on Saturday, June 9th, 2012 at 8:59 pm and is filed under General Outages, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

123 Responses to “Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 9 –9pm PDT(4 hours) Part 2: Cut Over(REVERTED)”

WTF are you serious!!!???? this is getting ridiculous! How many times per month can shithost go down or have an issue? Seriously the only reason they call themselves DREAMhost is that they DREAM to have more than a 10 percent up time. Here i have more than one dedicated and about 5 vps’s. This is absolutely UNACCEPTABLE.
I am really starting to consider other companies to hand them my money (such as a smaller hosting company that can at least provide some BASIC support. Lets see how about rackspace, softlayer, hostgator, GODADDY for gods sake is better than this and i left them over three years ago to avoid this kind of rubbish.

GET IT TOGETHER SHITHOST

or you will lose me as a client for GOOD

P.S also remember for every good experience a person tells on average ONE other person.
For EVERY BAD experience they tell 20 ( i personally think more like 100+ due to yelp and many other social media outlets.)

AGAIN ALL SITES ARE DOWN<

WHAT THE HELL IS THIS?

DREAMHOST IS CLOSING OFF BUSINESS?

WHY ITS KEPT HAPPENING EVERY DAY.

IT WILL BE SOLVED TODAY OR WE GO FOR SOME BODY ELSE?

Regards

from:
haroon ahmed
e-solbiz team

All is down, the panel as well.

Wow…
Two times I try to get to my websites, two time it’s down.

How many time per day can you cut off the servers ?
I was going to pay for my bills, and the panel is down.

Are you really serious ? No, I mean, really ?
You cut it during the night in the us, that’s ok, but remember that in europe it’s the whole _morning_. These days you cut my websites half of the time people want to access it !

You should give at least some weeks of free hosting for that. Because right now I pay for nothing. I might as well cut my account and go elsewhere like I should have for a while.

I agree with Jeffery.

I have used many hosting services, but had never seen anything like this case. This may be my greatest damage

Sites are down again. been like this for a while. upsold me to private servers. what a joke.

This really is a disgrace as a dedicated server user I have seen so much downtime over the last 6 months. This company is rapidly going downhill.

I must admit it is rather unpleasant ! and much to frequent for the price we pay, is it not ?
We shall see this time how long it goes.

I have a conference tommorrow and I was planning to make a demonstration to my colleague, please do not turn me out right in the middle (Seoul time from 2.00 Pm to 3.00Pm).

Good luck for this new issue.

PS: Can we have a discount on the next bill ?

This is a major day for me.

Please can this be fixed ASAP

David
London

How come site is down now?

Still down, please fix and please update us with some truthful words

Feature suggestion for dreamhoststatus.com:

1. log in with username/email
2. big message that says “you are down, we know” or “everything is ok”.
3. THEN the details.

Instead of me having to read through 3 paragraphs about technical stuff I do not understand ;)

Also, are you guys aware that whatever you report on dreamhoststatus totally happens in delay in other parts of the world?

e.g. if there’s a 4 hour maintenance, our sites are down 9 hours after the maintenance is over (it’s been like this the whole 5 years I’m with DH). Would be awesome to have dreamhost status updaete on real time for us non-US users…

Hi

ALL SITES ARE DOWN AGAIN

ANY BODY FIXING IT OR NOT?

JUST TELL US ONE TIME AND IT WILL BE LAST TO SAY FROM OUR SIDE.

@ralph ANy updates please?

Pleasele us know something.. today we are media partners in a big event and site is not working.. people are going crazy

MY SITE ´S BEEN DOWN FOR ABOUT 12 HOURS. PLEASE FIX THIS PROBLEM ASAP OR I WILL BE FORCED TO MOVE TO ANOTHER HOST. IT´S BEEN BECOMING MORE THAN USUAL YOUR HOST PROBLEMS AND YOUR INCAPACITY TO SOLVE THEM ON TIME.

MY SITE ´S BEEN DOWN FOR ABOUT 12 HOURS. PLEASE FIX THIS PROBLEM ASAP OR I´LL BE FORCED TO MOVE TO ANOTHER HOST. IT´S BEEN BECOMING MORE THAN USUAL YOUR HOST PROBLEMS AND YOUR INCAPACITY TO SOLVE THEM ON TIME.

Any idea when we can expect to be back up? It’s been about 4 hours now. Thank you.

Still down. Anybody else?

Yeah me too Dean – no response from tickets

VERY FRUSTRATING

NO RESPONSE, I do not understand why these clowns can’t streamline processes and upgrade/move servers etc and have no downtime. Constant downtime, the only time there should be downtime is for unexpected SERIOUS events, not crap like constant planned upgrades. DREAMHOST your upgrades are obviously not working because your service and hosting is getting $hitter by the month!

Wow. I made a big mistake coming here. Every morning UK time, the service is either dead or some part of it is super slow(>100 seconds response time). It’s unreliable and unusable.
DH – be upfront and tell us when this pain will be over. I don’t mean today’s pain. I mean, will this all be fixed in a month, 2 months, 3 months? Or is this it?? Is this the standard we should expect going forward.
I’ve noticed that even the dyed-in-the-wool DH lifers have quit singing your praises.

ANY BODY LISTENING OR GONE TO BED? OH MAN COME ON ITS NOT SUNDAY HERE IN THE WORLD ALL ARE WORKING AND YOU ARE SLEEPING GET UPPPPPPPPPPP….

All my precious sites, for which I’m paying more than $50 a month, are again down. I lost counting how many times this happened in the recent months. Yes I know it’s not your blame, it’s God’s willing.

Anyway, I want to thank once again DH for choosing the middle of a working day here in Europe to make the repairs. So all of us foreign customers are considered second class customers. Ok, I got that and I’m ready to make my move.

I’m back. Good luck to everyone else

My sites have been down all week. FIX THEM!!!

Wait a minute…
I upgraded from VPS to DS a week ago, and everything became slow/intermittent, so, thinking that I needed mor RAM, I upgraded to a more expensive DS and now everything is down. Is this all a waste of my time and money? Should have I stayed with VPS and just maxed-out the resource slider? Or would I still be down either way.
Sorry DH, I’m usually not the type to post here, but I am extremely frustrated.

Site down all morning!

Oops, ya’ll broke something…

I think there is the ground for a class action here.
If there is any lawyer out there interested in this, he may drop me at letheos@gmail.com

June 8th, 2012 at 6:44 amTien Do Xuan Says:

Stop advertising 99% uptime if you have too many maintenance sessions like this. Be honest.

HI

WHAT THE HELL AGAIN NOW ALL SITES DOWN AGAIN.

DO THROW OFF YOUR HARDWARE IF IT IS NOT WORKING.

FIX IT OR REJECT SO WE CAN GO ANY WHERE ELSE, AS IT DOES NOT LOOKS LIKE YOU WILL EVER SOLVE IT.

Regards

ALL SITES DOWN

HELP

MANY TIMES… WHAT HAPPEN?????

WHAT THE HELL’S GOING ON! I’M DOING AN IMPORTANT UPDATE AND YOU GUYS DIE AGAIN!

Just wondering if you’re planning on providing an update as to why there’s been more downtime than uptime today. If this issue is completely resolved then could you report on the new one? I would love to report through the panel IF IT WAS UP.

My entire site is still down… Any updates?

Site down, database inaccessible, panel down so can’t submit support ticket.

Ditto. All my sites and DreamHost Panel seems to be down.

My website down again.

Webmail, DH Panel, and my website are down. -_-

An unplanned downtime notice posted here would be nice – I had to look at all the ranting in the comments to determine if it was my end or Dreamhost’ end. But I can’t even reach panel.dreamhost.com at the moment.

Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,Down,

Down here in California

Yep, everything’s down. Dreamhoststatus.com and the twitter feed are silent.

We’re down again!

It’s sad really. DH used to have such a good up-time record.

Oops, this is the older ticket. They actually did post this in the next status update below this one – why is Jun 07 showing up above Jun 08? Weird.

Unable to access the Panel, mail services down. Unable to submit trouble ticket.

All of my sites are down. What exactly is the point of status.dreamhost.com if the status is not updated?

Wow, all my sites are down again. I think this is the 8th time in about 3 months. Compare that to a few years with no downtime, back when Dreamhost was running smoothly.

This really is not good, not good at all.

This is getting really old really fast! Verio is looking a lot better right now…

Leaving your shit hosting company now fuck you fuck you fuck you, what am i sopost to tell my customers? what do they say to their customers?

everyone relax, dreamhost is on top of this problem.

this facebook post from the dreamtoast page should ease your mind.

https://www.facebook.com/photo.php?fbid=10151823329455553&set=a.10150365614430553.579632.69871400552&type=1&theater

I cant acces web mail, mail, and panel i need mail help :S

Hi there, my sites are down as well. Can we get an ETA on this?

fú…

Maybe Dreamhost admin decided that the source of their network problems is too many accounts. All this sporatic downtime should solve that.

DHost is down yet again. .. To the CEO of this company, please learn what is required so you can truly manage this company w/o constant problems… . Otherwise, step down and let someone come in who knows how to run it.

Nightmare host.

ALL company sites you are hosting are down, multiple email systems are down, as well as DH Panel.

We cannot file a support ticket since DH Panel is down, so please email me directly ASAP regarding when you expect these issues to be resolved and to confirm you will be crediting our account.

We’re long time customers but we will have no choice at this point but to take our business elsewhere if this is not resolved immediately and proper credit applied to our account. Thanks!

Hi there

While my website is up I can’t get into my dream host panel or my webmail.

Since I can’t get into my dreamhost panel I can’t contact support unless someone knows a way to do it from outside the panel.

Please help – sending me an email at this point won’t help since I can’t access my webmail.

Barbara

Site down. Dreamhost.com unacessible so no way to open a ticket. And of course dreamhoststatus says all is well.

One of my clients (their website given) is supposed to be the focus of a history channel review today. Unfortunately, my server and all access to panel are offline (along with my other PS sites).

I have to admit, this is becoming more prevalent than it has ever been in my six (almost seven) years with Dreamhost… are you trying to convince me to switch? :(

WOW sites, Email, and Panel are all down and not a single update peep on the Status site about any outages……. The way I look at it is, No News is NOT Good News!!!

Great. ALL my sites are down and the panel and mail. Biggest hosing on DH I’ve seen yet.

June 8th, 2012 at 11:03 amMelanie Figg Says:

It’s a bad sign when dreamhost.com won’t even load.
This needs to be fixed asap.

I cannot reach dreamhost.com, panel, one of mail server, and three sites….
What’s happening ?

Panel is down as well so we can’t even open a ticket.

Reporting major outage what’s the support email?

Can’t even access the control panel!

Dino, there will be no ETA, and even if their were it would be useless.

I’m just a lowly shared hosting customer, but they have had one shared server down since Tuesday. This morning’s update was that they were 40 percent finished, the day before they said more than one third finished.

you do the math.

My site is down. My mail server is down. Dreamhost.com is down so there’s no way to submit a ticket.

This is turning into the worst year I’ve had with any web host ever. What is the matter there with you guys? Previous years had been so problem free; now it’s a disaster at every turn.

fucking suck.

DHost is down yet again. .. To the CEO of this company, please learn what is required so you can truly manage this company w/o constant problems… . Otherwise, step down and let someone come in who knows how to run it.

What is going on? Cannot access dreamhost.com, panel, etc

Same here… everything is down again….

all my sites are down!!!!!

Looks like my site is back up.

Aaaand we’re back. For now ? :/

Back up…thanks

So if all my websites are down, yours is down, how is http://www.dreamhoststatus.com up? you must know that your own company’s sh*t so you host else where lol!

YAY! Stuff is working…….Mostly!

And they’re back! I somewhat appreciate the announcements and hopefully the benefits the hardware improvements provide – we did notice some slow-downs prior, looking forward to more improvements!

Panel is down and I can’t submit a ticket. Guess I’m not the only one, so I’ll just have some more coffee. Honestly though, I’ve had minimal problems with Dreamhost for the past six years. Usually silence indicates that they are working on something important and diverting their attention to status updates kind of takes a back seat for a while. My husband has been a network engineer for 25 years, I see this all the time. Hang tight, our sites will be up and running again soon.

It came back, finally.

Site is back up.

Ok, a fairly brief downtime that time.. i guess. Still, not instilling confidence.

http://www.dreamhoststatus.com stays up because it is hosted separately from everything else, for the case when a meteor strikes or California by aliens. That way we will always know when there is a problem. Except for when some dimwit trips over a cord, takes everything down and no one can be bothered to update the status site from their phone.

Just moved here from HostDime… Left them because of the high latency and large amounts of downtime. I really hope this isn’t what I should expect… The owners will kill me :-/

I’m at work and don’t have enough time to scan over everyone’s posts, but is everything okay now? I just tried and at least Filezilla appears to be connecting now. However, my webpage loads in only 1 out of every 7-8 tries, and when it actually does load, it is awfully slow. Also, I am clearly missing data from the last 2 days or so. Although I don’t mind (too much) that you may not have backed up the most recent data, what I am actually worried about is emails that may have been lost that I won’t ever know about because they possibly never arrived. Does that make sense? Does anything you guys did affect email? Sorry to babble.

This is it for me. I already started the planning to dump DH for another provider. This latest outage and the periodic outages not published on the Status page is the last straw. I’m going to Bluehost

This is beyond ridiculous, down for days.

72+ hours and all sites on my account are still down, and only auto-responders from dreamhost. can anyone tell me what’s going on? when will this be fixed? failing that, can anyone suggest a different web host?

I’m getting error 28 from storage engine on all my MySQL databases. This is horrendous, my users are currently losing their data as we can’t write to the databases.

I’m getting MySQL errors, but I’m sure it’s being worked on. Everyone quit crying.

ALL WEBSITES ARE STILL DOWN. i HAVE SUBMITTED OVER 7 TICKETS IN THE LAST 24 HOURS AND HAVE NOT GOTTEN ANY EMAILS OTHER THAN AUTO REPSONDERS SAYING THAT MY TICKET WAS BEING LOOKING INTO. JUST LAUNCHED A LIVING SOCIAL DEAL AND THIS IS KILLING OUR REGISTRATIONS!!!!!

PLEASE GET THIS FIXED!!!!

COME ON IT HAS BEEN ALMOST 24 HOURS!!!!!

Hello Bluehost!!

I am lucky (I guess)…this latest outage only affected my outgoing mail server, but it was a big, big pain to work around that. Perhaps I didn’t notice whether my site was down or not. May need to find a new host. Too many issues in the past 6 months.

My websites are also having MySQL connection issues. When I run an account test right when the websites go down, the results show problems with “Your jabber server is missing its configuration file”, even though I don’t use Jabber on any of my sites. Strange.

The majority of problems I’ve seen in the last 2 months have been related to the websites being unable to connect to the MySQL database server. It looks like I’m not alone with this problem.

AGAIN ANOTHER DAY STARTED WITH SLOW DATABASE CONNECTION.

CAN YOU JUST TELL US IT WILL BE SOLVED OR NOT?

SO ATLEAST WE SHIFT OUR APPS SOME WHERE ELSE AS ITS NOT LOOKS LIKE YOU WILL EVER SOLVE IT.

http://scriptmodifiers.com/esolprojects

EVEN PAGE DID NOT OPENS.

My VPS is down for more than 24h. Can you provide any information about when you expect your shitty service to be back on?

I’am a long time customer.
Every time you tried an upgrade, you finish with a disaster and I have to apologize to my customers explaining why their precious, money making site, is not working. This upgrade was supposed to solve the previous disaster you had with your network. I have enough of your “funny” apologizing e-mails.
You can go to hell.

Yeah this is pretty ridiculous now. I’ve been with Dreamhost for about six years, and always defended them despite the occasional glitch. But I’m now 3 days with all my websites down, and a copynpaste stock response to my enquiry, just telling me essentially to wait.

It’s not good enough. Websites aren’t a luxury, for a lot of us they’re our business, and every minute my sites are down I’m losing more people. Three days is absurd, although I read on here other people experiencing it for longer.

We all dutifully pay our fees, but you can’t expect us to pay for a service which has fallen apart and is now detrimental. If we are to stay with you, I’d be keen to know what compensation or reduction you will be offering to everyone who has been so categorically nuked by whatever you’ve done to our sites.

KNOCK KNOCK AGAIN DOWN.

Coming up on 4 days with all 16 sites on my account down. No response from DreamHost, and the statements in the Panel alert are not accurate.

I don’t want to sound hysterical about this, but there doesn’t seem to have been any progress made in the past days.

I put a code into my wordpress header for queries. This is what i get now.
116 queries in 3.344 seconds.

Before, it was below 100. I think it’s because of what’s happening with the servers. it sucks. site is slow. sometime the content on the site only half loads. It’s wierd.

I log into my mysql and i get

‘phpMyAdmin –

#1040 – Too many connections”

I take it cause of the servers?

Wow. Some of you people would bitch if you were hung with a velvet rope.

For crying out loud, people! DreamHost is trying to get you off of a tired out old system onto a brand-new up-to-date network and all you can do is complain and threaten to leave and take your ball with you.

Buck up, or just shut up and leave already. It is funny, but many of you have threatened to take all of your clients and leave several times in the past. Yet you are still here to complain. You claim that every outage you have is costing you (or your “clients”) thousands of dollars.

All I can say to that is horsepussy.

Dreamhost IS trying to upgrade their system. It isn’t like changing a light bulb. It takes months of planning and entire sections have to be taken down and physically moved. That entails some downtime for your website(s)

This is getting out of hand.
It shouldn’t take 5 days of having your website down to get it back up and to receive a reply or update from a company. My site has been down since Monday. This is out of hand!!!

All of my websites have been offline for more than a week now. I can’t even get my email/gmail for any of my websites either. Dreamhost issues are causing big damage to my little company and all the clients that depend on me to help make sure their websites are kept online and running smoothly. I’m very disappointed to be paying such a high price for Dreamhost’s inability to provide what I paid for!

DJ Allyn, I work in IT, the site I use is coming up on a week of downtime, I’m well aware the amount of planning that it takes to upgrade systems as I run a data center. If my folks had this kind of downtime during an upgrade I’d fire them all, piss poor planning, poor communication (status of this page says everything is hunky, yea right), and piss poor communication.

The only upside is dreamhost is cheap, and I guess you get what you pay for.

I agree with amazed, I am in IT and this is way to long to restore data to a server, the process is cleary flawed.

My site went down on wednesday and just came back up today (Sunday)…that’s 5 days being down!!!

If it was 1 day, maybe 2 days, I would accept that as the occasional hiccup of virtual hosting. But 5 days is absolutely ludicrous.

This is a disaster! How am I expected to run a business with the threat of this happening again?

I have been with dreamhost since 2006 and never experienced this before (which is not to say others haven’t). It’s easy to threaten to switch hosts, but this is actually not that easy to do. It’s a major pain in the ass.

I would much rather DS fix up their issues and give us a guarantee that something like this will never happen again.

Apart from this, DS has been a great service, has lots of features, easy to use and is great value for money.

But uptime is more important!!!

June 9th, 2012 at 9:26 pmata kavalci Says:

I think all of us has to support the company instead of complaining.

Because there is no alternative to dreamhost. All of us checked others and noticed that dreamhost is not only the best company, also very different from others! We love this place!

Personally, I am ready to pay more. I am not here because it is cheap. I am here because I hate other companies’ tricks and countless money traps.

I also live problems with too slow connections and frequent breaks. But lets be patient. I have full confidence on the staff, everything will be ok soon.

And a last word for the non-USA users: shut your mouth! hiring a overseas company is a luxury for us already, so it will be much more wise to be silent! (I am from Asia)

the worst hosting company.. wasted alot of money for all this nonsense..

My site finally crashed right now, this blows. I keep refreshing and I’m getting.

The connection was reset

The connection to the server was reset while the page was loading.

The site could be temporarily unavailable or too busy. Try again in a few
moments.
If you are unable to load any pages, check your computer’s network
connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

Also, it would be nice if they had the “CHAT” opened again like they used to.. Unsure why they took that down. hate submitting tickets. it takes them like 2-4 days to get At me.

This is just shocking, we’ve been down for at least 30 minutes now.

Should I redirect DNS and put up a “down for maintenance” page?

I’ve submitted a complaint.

Zzz… looks like my VPS went down.

liquidweb.com is the best hosting company in the world.. they have all the high end and powerful systems.. they also have the SSD hard disk type so your site will be damn fast.. the only thing is regarding the pricing.. dreamhost are using AMD and it sucks.. cant wait few more months, and then i will move away from dreamhost to LIQUIDWEB..

My site is DOWN. This is already the 4th time this week.

Hey everyone, Read the title of this article. It’s been up for days and you didn’t prepare? Quit crying.

Come on already!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network issue in Garland datacenter (RESOLVED)
11:06 am

Network issue in Garland datacenter (RESOLVED)

Posted (June 8th, 2012 at 11:06 am PST) by Brandon

We are currently experiencing network connectivity issues in our Garland data center. This is resulting in site downtime as well as panel unavailability. Our network engineers and system administration team are working to restore connectivity at this time. We will update this post with further information as it becomes available.

Update June 8th, 11:15am PDT:

This issue is now resolved.  If you are still noticing issues please contact support immediately and we will address your issue first.   We apologize for the service interruption and are happy to report that the fast reaction and hard work of our admin and NOC teams kept the downtime to under 20 minutes.

This entry was posted on Friday, June 8th, 2012 at 11:06 am and is filed under General Outages, Major Outage, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

24 Responses to “Network issue in Garland datacenter (RESOLVED)”

dreamhost is FUBAR

Looks like it all just came back up for me – sites, and panel.

Keeps going up and down for me.

Stuff is working again!!!! Thanks!

Is this why my customers can not view my webpage?

NEWSFLASH: It’s all been down for the last hour: Webmail, Panel, and sites!

Our email just came back up, but our website is down still.

Is it me, or does this happen at least once a month to everyone else?

Site it’s working now, but e-mail down.

Thanks for the update. Standing by.

Site still down. Happens way too often for small businesses.

Got to improve on this issue if you guys want us to keep our site through Dreamhost beyond one year.

It was a short outage, but 2 minutes after Pingdom sent me the alert, a client sent out a big marketing email pushing his site.

Remarkable timing.

Anyone know of a decent hosting company? The owners have had enough of this….

I don’t think we will do business with a company that doesnt give out a phone number….

Thank you for letting me pull my hair out for only ten minutes. I’ll go back to worrying about when this will happen again.

10 minutes equates to 23 phone calls from customers … Campground on a friday afternoon. We’re getting tired of this.

I have been a customer since 1998, but your service has become too unreliable. I’m leaving.

20 girl minutes, maybe. (“Are you ready yet?” “I’ll just be five minutes, honey!”)

Sites on ‘swarm’ all taking MINUTES to connect and serve each page. Customers complaining. Over 8 hours since EXTREMELY URGENT ticket sent – no reply.

I have been a customer since 1999 and, unfortunately, your usually good service is starting to slip recently.

Please get this sorted as soon as possible

Strange part is that ssh never went down for my vps…

Still Down? what is it with dh and fridays?

My site ihas been down for more than 12 hours! Cant believe this! I have submitted 2 tickets about this problem and still waiting for a response

My website and all my clients websites are down for many days again. Panel is not accessible. Even all our email (gmail) is no longer available. I’m seriously disappointed with Dreamhost’s inability to provide the service that we all paid for = instead creating damage to my little company that has passed the point of repair.

Hi

My sites are getting down again.

Can you pls guide me and confirm me how much time this whole procedure will take to get completed? Just tell me the time ???

1 days 2 days 1 week one month?

“asdasd Says:

Still Down? what is it with dh and fridays?”

This is because the kids in the college dormroom who are running this place go home for the weekend!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network issue between 2 of our datacenters (RESOLVED)
3:48 am

Network issue between 2 of our datacenters (RESOLVED)

Posted (June 8th, 2012 at 3:48 am PST) by patrickm

We apologize for any service interruption you may be experiencing. We are currently investigating an issue affecting traffic between 2 of our datacenters which is affecting connections from one server to another, most specifically in cases where a web server is in one datacenter and the MySQL server in another. This may show itself as failed MySQL connections on your websites.

Our Admins are actively working to resolve this issue and we will post when we have additional information or a resolution!

Update: This is now completely resolved! We were able to identify the issue as a configuration mismatch between the routers in two of our datacenters. Once identified, this was quickly resolved by our Network Engineering team.

As we continue our Network Improvement Series, we will be adding this to the checklist of items to be checked and double-checked to ensure this type of issue does not happen again.

If you are still experiencing any issues with your hosting services at this time, please feel free to Contact Support for further assistance!

Update 4:18am PDT: Our network engineers were called to resolve the issue with packet loss between two of our datacenters.  We believe the issue is resolved now, but we are still testing your sites to make sure there are no further unexpected issues. -PatrickM

Update 4:07am PDT: Our Admin team is continuing to thoroughly investigate this problem and are getting closer to resolving it completely. Please check back here for more details as we are able to provide them! -PatrickM

This entry was posted on Friday, June 8th, 2012 at 3:48 am and is filed under General Outages, Multiple Server Issue, Mysql Outages, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

8 Responses to “Network issue between 2 of our datacenters (RESOLVED)”

This is more than an issue of routing between data centres. I have many random sites that do not work at the moment and it looks like it depends on what their dedicated IP address is.

With a failed MySQL connection the website still loads with an error (or pages without MySQL load). But these sites are getting complete Network timeouts.

Finally some news. I thought I was loosing my mind because suddenly half of one of my sites wasn’t working.
After all these upgrades are done, it had better been worth all of the downtime this past month.

@Dave: I was not able to find an account associated with your email, can you provide me with an example domain, or please contact us from the Support > Contact Support page in your Web Panel?

@Jake: Your site is working at this time, as we may have resolved this issue! Wanted to confirm for you that your issue was related to the above outage.

My sites are still down. PatrickM I submitted a few support tickets (one subject Attention: PatrickM)

You can test by surfing through a proxy. There are many websites on my account that are not working at the moment.

Network Timeouts continue.

@Dave: Thanks! I have your ticket and I will be in touch shortly!

Still not working

@Ivan: I am having no trouble accessing the site you listed in your comment. Please contact Support from your Web Panel for more assistance, thanks!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 7 – 8pm PDT -10pm PDT(2 hours) Part 1: Staging (RESOLVED)
6:58 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 7 – 8pm PDT -10pm PDT(2 hours) Part 1: Staging (RESOLVED)

Posted (June 6th, 2012 at 6:58 pm PST) by Ralph

Our border router upgrades are now successfully complete, so it is time for us to move on to the core routers of our Orange County and Los Angeles based data centers. First up will be the Orange County data center at Irvine. We will be upgrading the core router in a two-part process.

1. Staging – Thursday June 7th, 8pm PDT- We will be installing the new router hardware and testing connectivity. Maintenance window will be 2 hours. This process will involve racking hardware, extending cables and testing the connectivity. We will not be routing any production connections through the new hardware. Service impact will be minimal, previous staging attempts have resulted in no downtime, but when making network changes we are always aware that service interruptions can occur.

Cut over to new hardware – Saturday June 9th, 9pm PDT- With the preparation and staging completed Thursday night, we will then be able to actually cut over to the new hardware and use it in our live network environment. This window will last 4 hours. The process will involve moving individual subsets of our network one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks. This means that downtime for individual customers will be limited. We do have several physical connections to switch so we expect to use the entire 4 hour window.

We thank you for your continued patience as we progress further in our Network Improvement plans. We will update this post for the Thursday staging maintenance 10 minutes prior to starting.

Update June 7th, 8:45pm PDT: We have successfully completed the staging portion of this maintenance. The work is set up for the transfer of our production network to our upgraded core hardware on Saturday. Thank you as always for your patience and support.

Update June 7th, 8:00pm PDT: Our network engineers are onsite and ready to rock and roll on the staging. Since we are only installing and linking hardware, we do not expect any service interruptions. We are merely preparing for the network cutover that will occur on Saturday’s maintenance window. We will keep our eyes open and you updated throughout the staging of this upgraded hardware.

This entry was posted on Wednesday, June 6th, 2012 at 6:58 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

19 Responses to “Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 7 – 8pm PDT -10pm PDT(2 hours) Part 1: Staging (RESOLVED)”

Hi I dont know if its anything to do with these changes, but could you please take a look at ticket #5133334 due to a server move I have been unable to access dreamhost hosted mail boxes for 6 days

Dear Dreamhost

Please let me know when hosting packages can active. All website in my account was inactive for more than two today.

my god.crash again??

All our websites are still down!!
Three days with this problem and no solution.
We’ve sent several support tickets, but there is no answer. I think is too much time to recover.

Please see ticket 5141066. This happens every time to my Rails site — there is middleware that has to be restarted whenever you fix/replace/reboot the server, and Dreamhost always seems to forget this. My site has been down for many hours since the “all’s well” notice went out…

php Mail has stopped working on all our sites hosted on dreamhost. This is major impact to our business as we do a lot of online reservations. This is happening since couple days now. I have opened 2 tickets but not response. please help.

same here as simon, almost two days of all sites on my account being down. lots of support tickets sent and many auto-responders.

All of my 50+ websites have been down for 3 days. I am losing clients and I need this FIXED. I’ve sent several tickets, tweets etc all with NO reply! You’re killing my business and pretty soon I won’t be able to recover. Get this shit taken care of ASAP!

This a Chaos, no answers, all down, no news, whats up??

down again.

My website has been down for 3 days and no response on any of my tickets. This is ridiculous. Very much thinking of moving on after being a dreamhost customer for over 10 years. Please send me an update asap.

I’m not even a customer of Dreamhost, but several of the sites I like to visit are. ALL of these sites have had crappy connections this entire week, with frequent timeouts, and prolonged periods of time where they are completely inaccessible.

I’ve been doing diagnostics and sending my results from my various computers and IPs to the admins of these sites to help them build evidence that YOUR problems are costing them time, customer goodwill, and possibly money.

At one site, forum traffic is down by nearly 2/3rds because people can’t get to the site.

I was thinking of moving my sites over to dreamhost when my current contract ends in August, but I doubt I’ll consider it now.

Hi, all sites down, please fix it as soon as possible!

June 7th, 2012 at 3:09 pmRui Alexandre Says:

Hey, all my fiztpatrick sites are down with erros such as:

“Infinite loop detected in JError”
“Database Error: Unable to connect to the database:Could not connect to MySQL”

What gives?

June 7th, 2012 at 3:15 pmRui Alexandre Says:

it’s back on. sorry.

Can you at least acknowledge you are reading this Dreamhost?

It’s completely unacceptable!!

Its a full 3 days of you trying to get bujumbura back in order. What is going on? Please give us an update!

How about instead of discussing this staging and update you guys get our websites back up and working. Can we have a status on that? Are you reading this? It been nice if our websites became a priority after being down for 3 days!!!

Update June 7th, 8:00pm PDT Our network engineers are onsite and ready to rock and roll on the staging. Since we are only installing and linking hardware, we do not expect any service interruptions. We are merely preparing for the network cutover that will occur on Saturday’s maintenance window. We will keep our eyes open and you updated throughout the staging of this upgraded hardware.

@Jeff – your issue is unrelated to this maintenance and we update the panel post multiple times daily to keep you updated on the issue. I am however assuming you are on bujumbura or another failed over machine. If you have not gotten an initial email reply and daily updates to the panel messaging please let support know. Those updates are our new way to communicate issues directly to customers, so if it’s failing you, we want to know so we can address that. We can not however get bujumbura up any faster unfortunately.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #3: Tuesday June 5 10pm PDT 4 hours-Part 3 Border Hardware Upgrade-Los Angeles Data Center(RESOLVED)
5:48 pm

Network Improvement Series Maintenance #3: Tuesday June 5 10pm PDT 4 hours-Part 3 Border Hardware Upgrade-Los Angeles Data Center(RESOLVED)

Posted (June 4th, 2012 at 5:48 pm PST) by Ralph

We will be continuing the Network Improvement Series Maintenance #3 Tuesday night, June 5th, at 10pm. We already have the new upgraded hardware installed and ready to go (that was completed during our Saturday maintenance window). We will now be routing traffic from our previous border router and on to the new, upgraded hardware. To minimize downtime during this maintenance, we’ll be moving our providers to the new router in steps. First, we’ll route all traffic through one provider and then move the remaining providers (without traffic) to the new router. Once that’s done, we will test the new router to make sure it’s handling traffic correctly. Then, we will switch all traffic to the transit provider that was moved to the new router. After that is finished we will then move the other providers (without active traffic) to the new router as well. Once that is complete we will have our new upgraded border installed and working which will allow us to proceed with Network Improvement Series Maintenance #4. Thank you for your continued patience and we will do our best to minimize the service impact with these changes.

Update June 6th, 2:30am PDT: Our upgraded border is now installed and working correctly in our Los Angeles-based datacenter! The final clean-up is finished and we have all sites working correctly and all network routes going through our upgraded border router. Please keep an eye out for the announcement of Network Improvement Series Maintenance #4, which will involve the core router at our Orange County datacenter.

Update June 6th 2:00am PDT: Most issues were resolved at around 1:00am PDT.
We are turning IPv6 on and then we will be resolving this status post.

Update June 6th 12:50am PDT: We made some progress on the clean up that will allow us to focus on more specific issues. For those of you I have touched base with in the comments and those of you who have contacted support, we are using your test cases to resolve the remaining issues we are seeing. We are focused on finishing this upgrade before the window closes at 2am.

Update June 6th 12:00am PDT: Most of the residual issues have been resolved. We are however still working on the remaining issues that spans between two of our data centers. This will affect customers with hosting in one data center and mysql in the other. We are getting closer to full resolution for all customers.

Update June 5th 11:40pm PDT: We are cleaning up reported issues as well as still verifying proper routing. The one issue we are seeing is with hosting servers being unable to connect to database machines. These are clearing up as we address them. Please contact support with issues or submit a comment, with domain information, so we can further assist you.

Update June 5th 11:10pm PDT: All providers are now running through our new upgraded border hardware. There was some minor service interruption (~5 minutes) due to BGP reconvergence once the transit took root on the new hardware. We are currently testing and resolving some residual issues, however most of our tests are routing properly.

Update June 5th 10:30pm PDT: We have just finished testing the first transit provider and are about to begin moving it over to the upgraded border hardware. To this point we have had no service interruptions and our network is stable. As soon as this transfer is complete we will update this post.

Update June 5th 9:55pm PDT: We are currently getting ready to start work on the border network upgrade. We’ll announce when work begins.

This entry was posted on Monday, June 4th, 2012 at 5:48 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

87 Responses to “Network Improvement Series Maintenance #3: Tuesday June 5 10pm PDT 4 hours-Part 3 Border Hardware Upgrade-Los Angeles Data Center(RESOLVED)”

Websites are down again. Any updates?

Working now. Hope it was just a temporary glitch.

Databases for site are down. Is this a known issue? Is there a status update on this issue?

Databases are all down. Come on guys. I thought this stuff was behind us.

database down.my god!

sometimes all websites all down

please hurry up

Guess I’m not the only one. It’s 8 PDT, maintenance is supposed to be at 10 PDT.

sweetbeettea.com and posttitsorgtfo.com i own both of them. im only posting this here cause i have no marketing budget. lol.. dream host! i wont leave you! im happy with my service. but please get this fixed. thank you. sweetbeettea.com

Hi guys,

had a few people contact me saying my website doesn’t load. When will the maintenance work be complete? It’s daytime in Australia and I got to get back online!

Thanks!

As of 11pm EST, all of my sites just went down with ‘500 Internal Server Error’ The upgrade is supposed to be Tuesday night?

please please hurry!

Anyone’s email down? .. . Also, I just reset my passwd to contact tech support but the online form won’t eat my passwd??

http 500 error..

500 Internal Server Error

FTP upload/download speeds are slow right now – 50k/sec. My websites were down for 50 minutes. Now they are back, but page loading speeds are sporadic and slow.

I’m getting a db connection error on one of my client’s sites. This is unfortunately the one site that cannot afford to be down right now.

“Error establishing a database connection”

Still : 500 Internal Server Error

all database are crash

My portfolio site is down as well with “Error establishing a database connection”

Have an interview in 12 hours, really need this resolved…

@kate
your site same with me.says “500 Internal Server Error”

Just ran an account test:

jabber service for XXXXXXX.com error Your jabber configuration directory is missing.
jabber service for XXXXXXX.com error Your jabber server is missing its configuration file.
jabber service for XXXXXX.com error Your jabber configuration directory is missing.
jabber service for XXXXXX.com error Your jabber server is missing its configuration file.
jabber service for XXXXXXXXXXXXXXXX.com error Your jabber configuration directory is missing.
jabber service for XXXXXXXXXXXXXXXX.com error Your jabber server is missing its configuration file.
jabber service for XXXXXXXXX.net error Your jabber configuration directory is missing.
jabber service for XXXXXXXXX.net error Your jabber server is missing its configuration file.

I don’t use jabber on any of these accounts.

The sites on my own acct are not down…but I have a lot of clients on Dreamhost…most of those aren’t down…but there is one crucial one that is down.

Someone at Dreamhost needs to look into the “flute:franculli” MySQL Server

FYI…the site in question on franculli just came back up

VPS server (PS40039) is down

Maintenance #1 started on May 9th.. now we are up to #3 (part 4). I do not see, anywhere, how many more are planned = when will it be done?

I believe ps103348. All of my sites are getting 102 errors and Cloudflare is giving errors too. Is this because of the maintenance or a VPS server config issue?

This is not right.

My business relies on your reliability. And so far there’s not much of it left….

What’s going on???

The mail server is down on a regular basis and creating a lot of problem. What’s the permanent solution?

often loses the available aliases.
Mail not received what is blocking us.
when will it end?
we’re in deep shit

Come on guys, almost 12 hours of down time and two support tickets with no response.

I`m getting really angry clients calling me. My ps34963 is still unresponsive!!

Would what your currently doing affect my blog posts? The past 2 days have seen my 2 newest posts on http://www.thefranchiseking.com/blog not show up-and then show up. On and off.

Please advise.

Pro Tip: Drop dreamhost

Current Outage: Shared web server ‘scalise’ being moved to new hardware (updated 13 hours 7 mins ago)

More than 13 hours waiting, this is bad

stop agian

Current Outage: Dreamhost web server ‘bujumbura’ being moved to new hardware

Site is down again, how about some notice! When will you be up!

Charlie

All my sites are down now. I’ve got a big event with lots of people checking the website. What is going on? This stinks. They were down last week, too.

Site is down. Please update.

Site has been down on and off for over a week according to one of my clients. It seems to be completely down for the last couple days. Any ideas? Getting the “error establishing database connection” error.

my vps does not work for 1 hour
please fix it now. my clients are calling me very angry

dreamhost is really shit

Dreamhost is pathetic

June 5th, 2012 at 3:58 pmJackie Thomason Says:

Good to know in advance but we weren’t notified about the outages during the day today (June 4 in the USA) nor is there any acknowledgement here that they happened. Just FYI: Information is a good thing. I shouldn’t have to try to explain to my clients without any information or any indication of when the problem is to be solved. Nor should I have to keep going to my site to see whether it is available.

where this site is hosted?

Because dreamhost server is down.

Site is down D:

@Myles, we are checking on your issue right now.

Error establishing a database connection on my site as well.

@Myles your site is is up now. MySQL connections issue. It was unrelated to this maintenance. @Mark, can you provide me with your domain or submit a support ticket so we can check?

All sites is down!!

Hey, I have DB issues too. Flute server. Already made ticket and tweet.

@Willy your issue is related to a server restore you should have been contacted by support that we are working on that issue. During those restores we touch base with you at least twice daily. This is unrelated to the maintenance. @D. we are investigating

ditto on the mysql errors. My humantorch sites can not reach my ds

@Jay your site should be working now, if not, please provide the domain name that is down.

I get Error establishing a database connection ???

My site is down.

@Teo we are investigating

@Ralph, ok thanks,
Please let me know what it was once you find the solution :)

@Teo This is related to the maintenance today. The route to your mysql machine is blocked. We are working on this right now.

@Erica we are investigating but this initially appears to be unrelated to the announcement. My support leads are working on this right now.

@Ralphs. Thanks, Ralph. I appreciate it.

@Ralph

Thanks man, there are a few sites mate and still unfortunately down, give you an example http://www.holidayrentalsorrento.com.au our db is sitting on ds1438. Sites on that work fine, just having issues with the humantorch ones

@ Erica Turns out your issue is related to the maintenance. The problems that are related are due to mysql database machines and hosting machines being across two separate data centers. Typically this is not an issue, but we are still cleaning up routing issues that are specific to these two data centers communicating. We should have this resolved in the next 30 minutes or so.

@Jay – this is definitely related. Connection issues between the data centers was not fully anticipated in this maintenance. We were more worried about the transit providers working correctly once switched to the new hardware. We are near resolution though and our network engineering team is actively working.

@Ralph, no probs we’ll hold on the phones are keeping us busy enough for now… Thanks for the update, you guys are truly transparent

My site’s still down, it appears the database server isn’t working. Re-booted it twice, still doesn’t work. The Apache server is up since I can SSH in and SFTP but my site’s broken without mySQL.

@Ralph – Thanks – I appreciate the help :)

@Baron Your issue is related to the maintenance. I apologize that the issue is not fully resolved yet. Network paths can be a beast to fully correct. I am using your site as another test case for our team. Thank you for your patience.

Hi

All sites are down pls fix it

@Haroon we will use your site for testing. We just got through a major hurdle. @Baron you are back up. @Jay you are back up now. @Teo you are also back up.

All my websites are still down, can’t access any of them

@Ralph Thanks for that,
Good luck

@haroon ahmed – issue ended up being unrelated. My leads are working on your issue right now. @luki – issue appears to be unrelated to maintenance, my leads are looking at your case right now as well.

@Ralph Cheers mate, this adds a whole new level of service. Cudos to Ralph for sharing our outage journey!

We’re up as you say and the phones are now silent ;) I’ll now close our support ticket

Jay

@haroon ahmed – Issue was definitely unrelated. An IP was hitting your apache hard. We had to firewall the IP and restart the apache. Your site is loading now.

@Erica – we believe there is a problem with the PS being crashed. Due to the network routing issues we had trouble connecting to it to see what the actual issue was. We are looking at it closer now. The actual server issue seems to be unrelated to the maintenance, but our inability to connect to it and troubleshoot was related.

@Jay – no problem at all!

ALL my sites are down… They’re showing a “XYZ.com coming soon” Dreamhost page… What gives? Is this part of the maintenance? They’ve all been offline for over 18 hours as of now.

Mine still seems to be down. When will it be fixed? My site ranking has already dropped down 2 pages on google.

@Susheel – not related to maintenance, please contact support immediately if you haven’t already. I have also alerted them to work on your ticket asap. @Erica – we are actively working on your machine now.

@Erica, you are up. Since you are using cloudflare, click, try the “click for live site” button. It was not working for me, but once our engineers told me it was working, I clicked that and it worked. You are up now.

My all sites are down

http://www.esolbiz.com
http://www.scriptmodifiers.com etc

pls fix its much trouble for us now

All of my sites hosted under my account (@francesdath) have been down for around 13+ hours with a variety of errors (e.g. 500, bad_httpd_conf, 403, occasionally also the ‘coming soon’ page).

Databases I currently can’t connect to using SSH, and have been up and down.

SFTP is variously down, opening in root (i.e. account directories are missing), or showing a very out of date backup.

I’ve sent three support tickets, and messages to DH on Twitter and no replies for any.

I’m on bujumbura, and puffer for sql.

Please tell me what’s going on.

@luki – your sites are up now, but please contact us with the issues you have been experiencing. We may need to investigate further. So just submit a support ticket with details any problems you are still seeing with your site. Your sites had a misconfigured .htaccess file.

@Ralph: Thanks again, Ralph! :D

@frances @Susheel Samuel – you are both on bujumbura and should have received email contact from us about the hardware replacement it is undergoing. If you have not received these emails please let us know via our support system and our staff will address your concerns there. We do already have your cases upgraded to administration.

All of my sites are still down. More than 20 hours!! All of them arewith the ‘coming soon’ page.

I have several clients complaining.

Please tell me what’s going on.

@Antonio – see my reply to frances and Susheel.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Servies Maintenance #3: Sat June 2 8pm PDT 4 hours-Part 2 Border Hardware Upgrade-Los Angeles Data Center (RESOLVED)
7:00 pm

Network Improvement Servies Maintenance #3: Sat June 2 8pm PDT 4 hours-Part 2 Border Hardware Upgrade-Los Angeles Data Center (RESOLVED)

Posted (June 2nd, 2012 at 7:00 pm PST) by Brian@dreamhost

We will be attempting to reschedule Network Improvement Servies Maintenance #3 in multiple parts in order to ensure the successful upgrade. A summary of the 3 part process is as follows:

1: (completed during the Wednesday maintenance window) Moved network routes to route around the hardware we will be replacing to minimize service interruptions.
2: (Scheduled for Saturday June 2nd)
Connecting upgraded hardware and verifying it is working as expected.
3: (Scheduled for Tuesday June 5th)
Moving transits and going live on the upgraded hardware.

This part of the maintenance will occur this Saturday June 2nd at 8pm and will last for 4 hours. There is expected to be no service interruption during this maintenance as we will just be connecting the new hardware (on the backend) and verifying it is functioning as expected. No traffic will be routed to it from the outside. Although this is not expected to be customer facing, this is a network change and the nature of these changes can produce unexpected service interruption. This maintenance is laying the foundation for Tuesdays Part 3 maintenance where we will be finalizing the cut over to our new upgraded hardware. We appreciate your patience during these maintenance windows.

Update June 2nd 9:15pm PDT: The new hardware has been successfully connected and there were no service interruptions. Please keep in mind that Tuesday we will continue with Network Improvement Series Maintenance #3 where we will actually move the transits over to the new hardware.

Update June 2nd 9:00pm PDT: The new hardware is now successfully connected to our production network. Service has remained stable and unaffected during the installation. We are now monitoring the network to make sure there are no unexpected changes. So far everything is working correctly and looking great.

Update June 2nd 8:30pm PDT: We are still in the process of connecting the new upgraded hardware to our production network. We are in the process of making the connection to the new border, however we are not quite connected yet and are in the process of ensuring proper operation. There has been no service interruption at this point of the maintenance.

Update June 2nd 8:00pm PDT: The new hardware was pre-installed and ready to go. We are now starting to connect the new upgraded border hardware to the production network. The chance of service interruption is low at this point, but changes like these do have risks. We will be monitoring the transition closely.

Update June 2nd 7:55pm PDT: The team is ready to start on the maintenance. We do not expect there to be any service interruption during the new hardware installation, but with any change, there is a chance of interruption. We will be monitoring our service to make sure there are no issues, or minimize the impact of any that arise, and keep you updated on our progress.

This entry was posted on Saturday, June 2nd, 2012 at 7:00 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

19 Responses to “Network Improvement Servies Maintenance #3: Sat June 2 8pm PDT 4 hours-Part 2 Border Hardware Upgrade-Los Angeles Data Center (RESOLVED)”

My site is running slow since 7am this morning, no one has contacted me back after opening two support tickets. Is anyone live at DH?

June 1st, 2012 at 7:30 pmDave Thomas Says:

All sites running slow all day (and intermittently all month) too.

My sites are running faster than before the failed hardware upgrade. Fairly light weight websites, most pages loading in 0.3seconds, considering pinging dreamhost is 0.2s, since i’m halfway across the world, I’m pretty happy.

Also for the last couple days having multiple times on multiple websites “Error establishing a database connection”. This to is intermittent and getting frustrating. I can deal with this if this is all going to be corrected after all of the upgrades. Can someone from DH just let me know if there are some issues randomly and they will be resolved? It would make me feel better than saying everything is running fine. I have multiple hosting providers but prefer the awesome panel, even though the other hosts I use RARELY go down.

WTF?!? My DreamHost VPS is down, MySQL down, Web Panel down!

All down once again…

Yep, all my sites are down too. Same with the http://panel.dreamhost.com

All my sites are down, too! Of all the days, this is a very important day for my sites D:!

Someone needs to turn-on the Bat Signal.

Everything is down.

DH control panel and all emails down

Back up.

y u no have website stay up very much in 2012?

All down again…

And down again. Seriously what the hell.

..and down again.

Down again

Yep — site, mail down. It’s June 2nd, but maintenance isn’t supposed to start until 8pm and there isn’t supposed to be any downtime. As soon as the several guys who are coding part of my site are finished, I’ve got to move.

Looking like Media Temple might be the way to go. 3 times as much per month (based upon what I’ve paid to Dream Host for 2 years of fees in advance) but that’s a drop in the bucket compared to the inconveniences I’ve had to endure.

website down.. losing money with every passing minute. nightmare host : never again..

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Connectivity Issues (RESOLVED)
12:24 am

Network Connectivity Issues (RESOLVED)

Posted (June 2nd, 2012 at 12:24 am PST) by patrickm

We apologize if you were affected by a brief service outage this evening, which occurred between 11:40pm PDT and 12:06am PDT which affected servers in some of our datacenters. We are currently investigating the cause of this outage and will be posting more details when we are positive we have found the root cause and implemented a solution!

We are spot-checking services at this time, but it appears that service has been restored for all customers. Please Contact Support if you are still experiencing any issues with your hosting and we will be happy to help!

Update Jun 2, 2012 1:51am PDT: We are happy to report that things have remained stable since the outage earlier and we are resolving this active post. Currently our investigation points to a hardware issue. We are continuing to monitor hardware and have verified redundant hot spares are in place and functional.

As always, if you are experiencing any issues with your hosting, please contact support for further assistance!

This entry was posted on Saturday, June 2nd, 2012 at 12:24 am and is filed under General Outages, Major Outage, Multiple Server Issue, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

39 Responses to “Network Connectivity Issues (RESOLVED)”

Webmail is not working at all.
Have tried repeatedly to get in.

in firefox this message:

The connection was reset

The connection to the server was reset while the page was loading.

Webmail completely fubarred.

@JM @BW – We apologize for the WebMail outage that occurred following the posted outage above. That service was restored around 1:20am PDT. Please contact support if you continue to experience any more issues with that service!

panel just went splat

502 Bad Gateway on my site, and dreamhost panel seems to be down.

Wow, seriously?? Please fix the panel.

how are we supposed to contact support when the panel is down.. 2nd time in a few days.. :-(

Yeah.. and since I’m using Cloudflare, my entire site is a 502.. I can’t to squat until they fix it so I can change the name servers. Bleh.

Panel unavailable 11.47 GMT

Panel down – nothing on @dhstatus, no official post here…?

DH – “if you are experiencing any issues with your hosting, please contact support for further assistance!”

How do we contact DH when the Panel itself is down?

Hemant: email support@dreamhost.com (preferably from the email address associated with your account)

I cannot access to https://panel.dreamhost.com

Help pelease.

Chris: Thanks. Meanwhile, please can you indicate the reason for the Panel outage? And when will access be restored?
Regards

you cannot send an email tu support out of the control panel, bu we cannot access to it!!!!

Hemant: I’m a DH customer like you, sorry – just trying to help :)

what hell is this message from the mail server?

can’t create user output file. Command output:
procmail: Couldn’t read
“/mnt/thisbuds/vol/boot/homie/adjudicated/x9500099/.procmailrc” procmail:
Error while writing to “/var/mail/dhmail”

Chris: Oops. So sorry. Thanks, though. :-)

Hi,

https://panel.dreamhost.com/ is down.
Sending an e-mail to support will not work as their is no existing case.
Reply I get when sending out an e-mail to support is that I should submit new issues via https://panel
Catch22 anyone?

Antonio: I’m having mail problems today as well – no inbound email coming in. Have raised a ticket but no reply from DH yet. Where are you seeing that message?

Direct email, about the Panel-outage, to support@dreamhost.com returned the following. Note the subject line. :-)

[quote]
SUBJECT: Your support message was NOT received!‏

We’re sorry, but your email was unable to be processed by our automatic
support system:

We could not find the open support case you were replying to, so
your message could not be tied into our system. If this was a reply
to an EXISTING issue, please be sure to keep the:

[blah 123456]

tag in the subject line of your reply! We use that to keep track of
what case your message is associated with.

If this is regarding a NEW issue, we must ask that you submit it
through our web panel at:

https://panel.dreamhost.com/?tree=support.msg

(You can log in with this email address and your panel password.)

All messages submitted through that form will be received!

Our apologies for the inconvenience,
The Happy DreamHost Support Team!
[/quote]

Would be great if https://panel.dreamhost.com/ was working :-/

No panel.dreamhost.com for me either. Email when fixed please

Direct email, about the Panel-outage, to support ( at ) dreamhost ( dot ) com returned the following. Note the subject line.
Customers are pssed with the frequent outages, while DH auto-responder signs off “The Happy DH team”. WTH?!?

[quote]
SUBJECT: Your support message was NOT received!‏

We’re sorry, but your email was unable to be processed by our automatic
support system:
We could not find the open support case you were replying to, so
your message could not be tied into our system. If this was a reply
to an EXISTING issue, please be sure to keep the:
[blah 123456]
tag in the subject line of your reply! We use that to keep track of
what case your message is associated with.
If this is regarding a NEW issue, we must ask that you submit it
through our web panel at:
panel ( dot ) dreamhost ( dot ) com
(You can log in with this email address and your panel password.)
All messages submitted through that form will be received!
Our apologies for the inconvenience,
The Happy DreamHost Support Team!
[/quote]

Panel is back up!

Hemant: Sorry about that. Works for me when I email support@dreamhost.com. Maybe your email address isn’t the one you signed up with DH?

Chris: Hmm. That’s possible.

(I had originally registered my account with the ID I emailed from, now. Later, I added another ID. Thereafter, they send billing info etc to the original email ID, and newsletters to both email IDs. But I think I should have sent from the new ID. Quite weird! I will try that, the next time something crashes, again. :-))

Panel’s back up!

My website is still down since all of this started. Please get it back online, this is my company here!

Grr. There are issues on homiemail-sub3. Inbound email is bouncing back to senders with this message:

can’t create user output file. Command
output: procmail: Couldn’t read
“/mnt/thisbuds/vol/boot/homie/adequately/x9431283/.procmailrc” procmail:
Error while writing to “/var/mail/dhmail”

PS. When did Live Chat get taken away from the Panel? I noticed it has been down for the last few months, now it’s vanished completely?

wordpress are dashboards are still dragging pretty slow. please tell us this is still being fixed.

My wordpress sites are down completely, but other bits of my domains are up. Anything still going on?

For the last few days, it seems all my sites are slow to load – some over 1min 50 secs. Some fail to load and I receive a CloudFlare screen. Tried turning off CloudFlare thinking that is the problem, but to no avail. I guess I’ll report the problem again…

Steve you’re for sure not alone – I think alot of sites are slow.

We have several WordPress sites and the front ends seem slow, and the dashboards are dragging painfully… been happening the past day to day and a half.

We contacted support but haven’t heard back yet so we hope they are working on it :\

Seriously. Support responded to us and told us that the site slowness is specific tou OUR websites and is a problem with OUR scripts only and that we were using too much memory on the shared server (and just a note, our databases are actually on VPS guys).

But are you kidding us???

“Our sites” and “Our scripts” were not having issues until – oh the past 24 hours when everyone else started having issues. Coincidence??? Hardly.

This is our biggest struggle with Dreamhost – If you guys have problems, and you address them in timely manner, and communicate openly – great, we all move on. We have put up with countless outages the paast 6 months and have not complained – we just ride it out.

But when you guys say the issues are fixed – but our sites are still slow, we do what you tell us to do and we email support. But then you respond by blaming your customers websites for server issues with the “process watcher” email. This leaves us with no solutions because the isseus are NOT our websites – so we can’t troubleshost and fix something that is not the real issue.

Please just address the server issues and give people a break.

All my websites are down!!! Your Dreamhost status website said all is fixed… Really?

I created a support ticket, and have not hear back from you.. not even an automatic email confirmation.

You guys are lucky to have a “slow” site. All my sites are down. I paid for my vps just so that I won’t have to share any resources. Now I am swapped with calls, and no response from dreamhost support.

We are spot-checking services at this time…

I assume that’s in addition to using Nagios automated checks.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #3 Wednesday May 30 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (RESOLVED)
5:17 pm

Network Improvement Series Maintenance #3 Wednesday May 30 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (RESOLVED)

Posted (May 25th, 2012 at 5:17 pm PST) by Ralph

We will be attempting to reschedule our previously postponed network maintenance to Wednesday May 30th. This maintenance is dependent on provider upgrades so we will have to set a “Go” or “No Go” on this particular maintenance. The decision to “Go or No Go” will be on Tuesday May 29th, however we are fully committed to perform this maintenance if our provider upgrades are complete.

We will be conducting planned maintenance on our Garland border router starting at Wednesday May 30th at 10pm Pacific Daylight Time (PDT). We expect the maintenance work to be completed by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through May 31st 2am PDT.
Here is the sequence of expected events during maintenance:
– Maintenance in our Garland data center will start between 10 – 10:30pm PDT, resulting in all web hosting services in that data center being offline;
– A new border router that has been set up and prepared for operation will be connected to the network and tested, with a potential duration of 45 minutes of service interruption, for the testing to be completed;
– Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Garland and all web hosting services will be back online; we expect this to occur by 11:30pm PDT, however it is possible that network services will not be restored until May 31st 1am PDT.
– In the event that the new router does not pass all tests by midnight PDT, we will restore services on the current network router hardware, so that in any scenario, all network services are restored by 12 midnight PDT.

Update Thursday May 31st 1:50am PDT: The first step, which involved rebalancing our traffic, was a success. We are however rolling back the hardware upgrade. We were not 100% comfortable with the progress that we made during the maintenance window. We will be rescheduling the hardware part of the upgrade as soon as we confident we can get through the issues we noticed today. All services are now restored. Thank you for your patience during this maintenance window.

Update Thursday May 31st 1:20am PDT: Some services are not available right now. We are attempting to finalize this change by 1:30. Thank you again for your patience during this maintenance window.

Update Thursday May 31st 1:10am PDT: We realize that service is down again and we are restoring now. Service will be restored within 5 minutes.

Update Thursday May 31st 12:50am PDT: We are live on the upgraded hardware. We are still working to make sure everything is in order and working as expected.

Update Thursday May 31st 12:42am PDT: Final attempt is starting now. Our cutback process has been flawless on getting us re-established so we are trying one more time. We apologize for the multiple attempts and we are limiting downtime to a max of 10 minutes. Panel will be inaccessible during our final attempt. Please bare with us as we try to finalize Network Improvement Series Maintenance #3.

Update Thursday May 31st 12:28am PDT: We have one more configuration change to push through so we will be going dark for roughly 5 minutes. Our rollback is confirmed to work so we can rollback quickly if we determine there are still further issues.

Update Thursday May 31st 12:18am PDT: We are currently dark in the data center and are working on restoring service. Please note that panel is also down. Apologies that this was not an announced expectation of the change. Please hang in there and we will have service restored shortly.

Update Thursday May 31st 12:10am PDT: Okay we are switching to the new hardware now. This is our second attempt. If this one fails we will cutback immediately. Our cutback plan is already in place but we are expecting this to go smoothly. We are expecting less than 10 minutes of service interruption.

Update Wednesday May 30th 11:50pm PDT: We have yet to experience any outages. Hardware upgrade has not occurred yet as we wait for further route normalization. We were previously ready to make the switch when we noticed some route issues. Once that is cleared up we will switch the hardware. Previous hardware cut over was temporarily aborted.

Update Wednesday May 30th 11:15pm PDT: We are cutting over now, please standby for service interruption. We will do our best to minimize the impact of this upgrade.

Update Wednesday May 30th 11:00pm PDT: We are ready to upgrade the hardware. There has been no service interruption to this point of the maintenance. At the hardware upgrade we are expecting minimal service interruption.

Update Wednesday May 30th 10:40pm PDT: First step is complete however we are waiting for traffic patterns to normalize prior to proceeding with the hardware upgrade. There has been no service interruption during this first step.

Update Wednesday May 30th 10:20pm PDT: First steps of the upgrade are being executed as I type this update. This part of the update is not expected to cause any service interruptions. We are monitoring support as well as testing these changes as we make them. Once step one is complete the actual router hardware for the border will be replaced.

Update Wednesday May 30th 10:05pm PDT: We are roughly 5 minutes from making our first changes. Final work for the first step of the upgrade is still being completed before making changes live.

Update Wednesday May 30th 9:50pm PDT: Our network engineers are on site and we are getting ready to roll out the maintenance. We will be sure to keep you updated as we progress on this upgrade.

Update Wednesday May 30th 6:00pm PDT: Tonight’s maintenance is going to be performed as scheduled. We will be starting maintenance at 10pm but do not expect possible service interruptions until roughly 11pm. We will be keeping you abreast of the maintenance with periodic updates starting around 9:50pm tonight.

Update Tuesday May 29th 8:00pm PDT: We will proceed with Network Improvement Series Maintnance #3 as scheduled for May 30th at 10pm. Early tomorrow we will be performing tests to ensure all fiber and facilities are ready to go. If there are any further issues deterring us from acting on this maintenance we will make that determination well before the 10pm start time. Thank you for your continued patience during these series of Network Upgrades.

This entry was posted on Friday, May 25th, 2012 at 5:17 pm and is filed under General Outages, Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

112 Responses to “Network Improvement Series Maintenance #3 Wednesday May 30 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (RESOLVED)”

As is always the case, customers have NFC what hardware or services are housed at “Garland DC”.

LakeRat — yep.. I hear ya… it seems as if the UI design and customer facing communication paradigm hasn’t seen innovation for years…

meanwhile, panel.dreamhost.com is down for me as of 1:32AM central time… anyone else having a similar issue? my sites are up… but the panel is down…

Firefox has the following suggestions:

https://panel.dreamhost.com/

The connection was reset

The connection to the server was reset while the page was loading.

The site could be temporarily unavailable or too busy. Try again in a few
moments.
If you are unable to load any pages, check your computer’s network
connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

PANEL CRASHED (RESOLVED)

it’s working again. for now.

How can I tell what data center my sites are in? The Dreamhost panel shows me the name(s) of the servers, but I can’t find where they’re located.

I just pinged one of them, and baku.dreamhost.com, and it’s IP address shows it as being in Syracuse, NY! (69.163.183.230)

It would be nice if I could know if anything of mine will be affected by the maintenance at the Garland data center.

webmail is down …. can not log into squirrelmail … pages / service
down on East coast – New York City

My website is down again. Honestly, why do I continue to bother with you guys?

Sites on mangano server have been down for hours, with only SFTP and shell access.

Everything is crawling:(

A lot of my sites on dreamhost having issues this morning. Very, very slow. I tried to login to Dreamhost control panel and that was really slow and never fully loaded.

It would be nice if they posted what was happening.

Same almost 50% loss to my webmail

Ping statistics for 208.97.187.193:
Packets: Sent = 37, Received = 19, Lost = 18 (48% loss),
Approximate round trip times in milli-seconds:
Minimum = 80ms, Maximum = 91ms, Average = 82ms

trace:
…..
8 50 ms 51 ms 51 ms 10gigabitethernet3-2.core1.den1.he.net [184.105.213.86]
9 75 ms 75 ms 75 ms 10gigabitethernet1-4.core1.lax2.he.net [72.52.92.38]
10 79 ms 79 ms 79 ms newdream.net.any2ix.com [206.223.143.83]
11 * 96 ms 76 ms ip-66-33-201-114.dreamhost.com [66.33.201.114]
12 81 ms * 82 ms roundcube.dreamhost.com [208.97.187.193]

site is going really slow, there is no new post. Mail server is going rough too. even dreamhost. site is not loading.. Please post info ASAP. can not go into control panel to post problems.. What happen?

If this is “not a way to contact DreamHost”, then how do we contact DreamHost? Everything is down!

Seeing MAJOR packet loss and can;t access the Panel right now!!!

May 28th, 2012 at 9:02 amHans Eriksson Says:

Sadalbari is incredibly slow right now.

Can’t get my staging site off the server via FTP and I got a client waiting. I really would appreciate if you fix this as soon as possible despite the fact that it’s Memorial Day in the US.

Very very slow!!!!

Up/download speeds are 10K/sec right now in FTP. Websites are take 8 to 20 seconds to even respond to a page load. Nice!

I’ve had more page load issues today (unreachable MySQL databases, “This website is offline” messages from Cloudflare, 404’s, 500’s, 503’s, 505’s) than I’ve ever seen in my life.

The tech team steps away for a day off and the servers come to a stand still. Awesome!

is very slow guys!

e-mail is very very slow right now.

slow, my sites are slow :(

For the first time, Canbits sites are affected.

Webmail seems to be down is this related?

Our website is down, too. I want to know specifically what is going to be done for us; most of our views come during the morning hours, so at this minute we’re losing money. Also, the only time I have to add new stuff to the site is mornings, hard to do when the site is down. Customer service should be your top priority, so again I ask, what are you doing for your users who frequently experience these problems? Refunds? Anything more than an apology?

Shockingly, things are down after a planned upgrade.

My blog pages are down.. and those getting hit are where we make our money.. when I try to log into the wordpress admin.. it says “you must upgrade your database”.. but I’m not about to click that..

I’ve submitted two service requests to dreamhost thus far.. no response yet..

Dears,

Our webmail is dowm… we just transferred our e-mail accounts to your server and you promised to activate the webmail within two hours, and 3 hours have passed without getting any response from you…
kindly respond to our complain as we need to activate our webmail service urgently.

Thank You

Yesterday we were missing order confirmation emails. I sent a support ticket through the web panel. Received an email reply “No mail is lost, we’re processing it all in the order they are generated on your server.” Late yesterday we were still missing around 30 emails so I contacted dreamhost again and received this reply “I checked on the emails sent from your webserver this morning, and some mail was lost.” So don’t believe them if support says the mail isn’t lost.

kindly update us with the maintenance status as we need to follow our business…

Arrgh, many sites throwing errors – angry clients.. Co’mon people!!

Could this be causing a 500 Internal Server Error for a php upload form submission?

@Rob No this would be an error where you can not connect to your site. This is network related, so the route to your site would have issues. If you are seeing php 500 errors, please contact support.

Thank Ralph!

It’s only an error that appears at the end of a file upload using cforms in wordpress. Everything had been working fine for months and now it shows this error instead of completing the process. No files are uploaded to the server. All other areas of our websites appear to be fine. We’ve posted a support ticket… Thanks again.

Hey Angry People, get glad not mad!
If you don’t like Dreamhost, stop your complaining! I’m thankful that they do keep this wonderful blog with outages and maintenance updates BEFORE, DURING, AND AFTER they happen. You rock Dreamhost! Now, go out there and find a more reliable host for the same price you’re paying DH! You simply can’t find one! I’ve been with DH for 3 years now, and I’ve only had minor outages, nothing major along the way. If you need a server to host mission critical data, host it yourself for “100% uptime!’! Then if it’s down the only one you can complain to is YOURSELF! Please, servers and all the other equipment are as fussy as people, if they don’t get what they need (cool air, power, etc.) then they get sick and stop working. DH can’t do anything about it. Next time, ask NICELY. DH has a world class support system, and being nice and saying “please” and “thank you” really helps!
– ISEAGE

Webmail was down but is back up again now…. they said 10 minutes and that seems to be about right.

Not sure if the switch is completed but I’m glad to see everything back up. Good job!

I guess we weren’t done yet…

down , timeout.

even panel is down for me…

Sorry all, thank you for contributing with comments. We are attempting a final hardware upgrade right now.

now i’m getting a 500 internal server error even when I try to access wp-admin.

@Rob it was definitely related. We have service back up now and your site is working.

Everything is down!!!

Hi there,

jakarta and peterli stil have problems… is this expected?

May 31st, 2012 at 1:03 amDave Thomas Says:

Hope things are going through as planned. Was up a few minutes ago, now back down.

Peavy just went down. Can’t access http://panel.dreamhost.com either.

its enough………………………..

May 31st, 2012 at 1:07 amChristian Robinson Says:

Panel down, email down – can’t report outage. Is anyone from DH monitoring this page?

the phone will be ringing any minute….

May 31st, 2012 at 1:10 amEls catalans tenim la cigala ben dura Says:

I have a VPS and my emails are down!!!

May 31st, 2012 at 1:10 amDave Thomas Says:

This is neither fun or funny.

@Ralph
Still getting the error though with cform.

Thanks.

All sites are down, panel is down, even status is down!

humantorch sites appear to be down, ds is happy and working well thankfully.

Email accounts created today though are also not working well. create but dont have permissions when trying to use them.

Panel also down for the count.

Thanks for the comments guys, your test cases help us assess the situation. I will update the status post with significant decisions being made. We are currently aware of the service interruption and service will be restored shortly.

Everything is down for me, mail, etc, it’s 3:15am.

No new services, but all accounts are down.

My God you are lot of whingers! I don’t know of a single ISP that is as transparent as Dreamhost for letting it’s clients know what is going on regarding problems and maintenance. My UK ISP tells me NOTHING about any problems .. period!

We pay very little for what is, on the whole, an excellent service from Dreamhost. I have been with them since 1999 and having seen what other ISPs are like with my other clients, some using dedicated servers at great expense, I have no intention of moving my many sites at this time.

Dreamhost are not perfect but NO other ISP is either. Dreamhost at least give the impression that they care about customer satisfaction and try to provide us webmasters with information when things go wrong.

Cut these guys some slack will you? This is a scheduled downtime intended to improve the service. At least it is being done during the US late evening, it is the start of the UK working day over here and if I can put up with the inconvenience then you can!

May 31st, 2012 at 1:19 amChristian Robinson Says:

Ralph: IMAP Email & Webmail is currently down for my users – nothing in the status post suggested that email services would be affected other than sites hosted at Garland? Is this related, or a separate issue?

Everything is so down. Sites, email, panel just about everything

May 31st, 2012 at 1:22 amChristian Robinson Says:

Hmm… Email appears to be down for DH internally too – haven’t receieved the usual auto-reply to the ticket I logged via email to support@dreamhost.com (sent via a backup email service not hosted at DH, obviously).

@R

Some of us work at night, this is an important time.
Also, noting that status is down.. does not detract from
respect for the work that the staff are doing.
Calm down.

SORT IT THE FUCK OUT FFS ALL MY WEBSITES ARE DOWN MY CUSTOMERS ARE GUNNA GO CRAZY I CANT EVEN TRANSFER MY DOMAINS BECAUSE YOUR SHIT WEBSITE IS DOWN

Service is still down ? nothing is working

Dreamhost should give lots of free service days, according to their 100% uptime guarantee. Sadly, that’s dreamhost, so they won’t give a **** for they own promises. If they would respect them, I would have free hosting for next 10yrs…

May 31st, 2012 at 1:26 amGarland Nightmare Says:

Someone in dreamhost.com is able to change a router that fucking bad datacenter, or should we wait for the idiot who sent us will face wet himself from fear?

I have nothing. Sites, mail, panel. All down. Chop chop Dreamhost…

All web hosting services would be affected. I apologize on the use of the Garland Data Center name since there is no reference point for you. If you are down due to network availability you are affected by this, this includes database access for certain sites as well. Email and panel are down as well. We are working to avoid having to do another window for the same upgrade so we are utilizing the majority of the 4 hour window we specified. We try not to use the full window, but there is urgency to get our network hardened for your stability.

FUCK THIS SHIT

All of my sites (and even dreamhost.com itself!) is down from the UK. It’s 9:26am here, start of the business day.

@sam

And the outage appears to have even affected your caps lock key. Weird…

Service, status.. freudian slip.

Please keep us updated on the progress. x

Retiarius>> Is it “dreamhost at least give the impression that they care about customer satisfaction”, when I submit ticked at 0:30 that my sites are down (that was yesterday, not today), and get response at 7:30, stating that now everything is running fine and next time I should submit ticket as soon as I notice the problem.

Do you think 7 hours response time is lightning fast?

STFU JENS IM GOING TO LOSE MY FUCKING BUSINESS LOSE MONEY AND THEN HAVE TO GET A BOSS AND BE A FUCKING AVERAGE JOE

Everything up for me? Dont know what you guys are bitching for…

May 31st, 2012 at 1:31 amWilliam Xavier Bobberson Says:

I just ripped a HUGE fucking bong hit and blew it in my cat’s face. That little bastard didn’t know what hit him. While he was stumbling around the room I logged on to Dreamhost and BAM, my shit was down. Now I have to shit. Where’s that cat?

dreamhost out 3:28 AM ……. pppffff

May 31st, 2012 at 1:33 amChristian Robinson Says:

Ralph – thanks for confirming details, this is exactly the type of status update we need. Sit tight everyone.

aaaaaaaaaand, we’re back. GW Dreamhost

May 31st, 2012 at 1:33 amChristian Robinson Says:

(Hosting back up…)

why are you on here then brian ffs ?

I’m trying to figure out exactly what your shit and that cat have to do with each other

hosting always go back up for 1 fucking minute

Oops! Google Chrome could not connect to nightmarehost.com

May 31st, 2012 at 1:36 amWilliam Xavier Bobberson Says:

Well, I plan on firing off my power dump and using my cat to wipe away the evidence. Don’t worry though, the previously mentioned bong blast killed the little fuck. Time for plan B. I’m going to go shit on the floor of a public restroom.

Hi

Please can you get the servers up and running ASAP! We are at a conference trying to promote our website and the server is down still!

“We are expecting less than 10 minutes of service interruption.”

heh… all I know is that was posted an hour ago, and currently the domains I have hosted with DH aren’t resolving… I think I’ve figured out who is hosting DH’s data center… AOL! pfff… just kidding.

yeah… it sucks if you have mission critical apps running on a virtual server, and you don’t have any contingency plan in place… but… that’s what you get for under $10/month, I don’t care who you host with…

May 31st, 2012 at 1:38 ammotherfucker Says:

Yawn… DH is down again – what’s new? The other day a customer called me, very alarmed: “Hey, my website is up: something’s wrong?” Fuck you DH

mail down, webmail down

web panel back up after 20 min outage

Everything is down, mail, sites, panel …. here in europe at 10:38 am (1:38 pacific) in the full business time without mail :-(

im gunna kill a cow

All my sites are down in the UK also – one of them to support a campaign that has been 6 years in the making and which has resulted in 9 directors going to court, one of them is in the dock right now… just sayin.

but if the cat’s dead isn’t it easier to wipe shit with? i mean what with the relaxed state, and not getting scratched while shit-wiping is pretty underrated. once rigor mortis sets in you can even use it to scrape off last week’s shit too

right thats it im suing you if I lose any money or business from this. luckily my dads a lawyer so I dont have to pay for 1

Service goes down once in awhile so whatever… after I thought that it went back up, I spent an hour typing out an article, hit submit and now all that shit is gone when your service took a dump on itself again. Way to go jackasses. Now I have to sit, wait, and lose sleeping till you get everything back up again. This is bullshit.

Come on guys! need to work.

I really love how many trolls and spammers come to this place.

last week 123 reg was hacked by the fucking chiness and 2 million websites gone! if it happened to be hackers then fuck of your not even good, just good guessers

All your sites are belong to DUST! what’s going on? Everything is down, including Dreamhost panel for the last hour. Dreamhost, you are singing off-tune lately. Could you please at least add an update on this page, so we could at least whine in the right thread?
Thanks!!

We’re up bitches.

May 31st, 2012 at 1:48 amme.no.smile Says:

Soooo.. two hours into the ‘fun’ so far.. first it was total snafu, then somewhat back again but now all fall down again at this moment = nothing ‘on’ here.. +_+

Aside from the world clock mucking up daytime action this side of the world.. do hope DH gets this sorted, even their own site returns 404, in the ‘proposed’ 4hr window.

Five+ years here with you guys and we ‘pay’ for VPS services…. #_#

I find the overall service to be good, in spite of the outages that occur too frequently for comfort. However, doing major maintenance on the last day of the month is bad timing; many of us have a monthly newsletter and update to put out at this time. Could DH try to do these things away from the month end?

LOL, you guys are hilarious.

You pay a low amount for what is a very good service.
Dreamhost gave prior notice of this possible downtime… what more do you expect???

If this really is a huge problem for you, then go find a hosting service that guarantees 100% uptime.
I doubt any of you will though, because if that was of critical importance you wouldnt be using dreamhost in the first place!!

Sounds like you guys need to spend more time out in the sunshine.

@Elliot is sooooooooooo right… If you don’t have redundancy plans for your hosting, don’t b*tch about your problems because that’s YOUR FAULT you don’t have any redundancy plans. You get what you pay for. And also, Mission Critical apps shouldn’t be ran on $10 or even $20/month hosts… they should be ran on dedicated servers that range $50+/month… You get what you pay for!

we’re up and running over here for humantorch again.. cheers DH… hopefully Im not too premature in celebrations.

wooooooooooooooo now time to transfer my domains and get the hell away from nightmarehost

Sam: Shut it. If you want to be taken seriously, start talking like a human.

What is the most annoying thing is that they don’t have a spine and nobody want to take action and do a roll-back. When you read their updates it’s always: “… if it is not working in 10 min. we will roll-back.” then 20 min later: “… this is the last try …”.
This just extended the planned outage over several hours and really cuts into European productivity time! The only right thing would have been to reschedule to a later date!
Dreamhost you make yourself look like a bunch of amateurs!

And…I’ll remember to define myself next time so I’m not confused with anyone else.

Back up. Thanks guys. x

Seems like quite a few e-mails were not receive during the downtime. Will they be delivered now that it’s back up?

Ok, mail received, all’s good. Thanks!

jakarta & peterli down again…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #3 Wednesday May 23rd 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (POSTPONED)
9:50 pm

Network Improvement Series Maintenance #3 Wednesday May 23rd 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (POSTPONED)

Posted (May 23rd, 2012 at 9:50 pm PST) by Ralph

We will be conducting planned maintenance on our Garland border router starting at Wednesday May 23 at 10pm Pacific Daylight Time (PDT). We expect the maintenance work to be completed by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through May 24th 2am PDT.
Here is the sequence of expected events during maintenance:

– Maintenance in our Garland data center will start between 10 – 10:30pm PDT, resulting in all web hosting services in that data center being offline;

– A new border router that has been set up and prepared for operation will be connected to the network and tested, with a potential duration of 45 minutes of service interruption, for the testing to be completed;

– Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Garland and all web hosting services will be back online; we expect this to occur by 11:30pm PDT, however it is possible that network services will not be restored until May 24th 1am PDT.

– In the event that the new router does not pass all tests by midnight PDT, we will restore services on the current network router hardware, so that in any scenario, all network services are restored by 12 midnight PDT.

Update May 23rd 10:30pm PDT: We have made a decision to postpone todays maintenance. During the final steps before the upgrade, our confidence in our transit connectivity was not at 100%. After exploring all possible options, we made the decision to postpone this upgrade, and keep your service running, as we assess the situation further. We expect this maintenance to be postponed till next Wednesday at the earliest. There was no service interruption due to any working being done tonight.

Update May 23rd 9:50pm PDT: We have been working hard preparing for the transition tonight and we are near ready to execute the hardware upgrade. In about 10 minutes we will begin work toward making the hardware switch. We will notify you with consistent status posts as we progress through the change. Thank you for your patience during this maintenance window.

This entry was posted on Wednesday, May 23rd, 2012 at 9:50 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

21 Responses to “Network Improvement Series Maintenance #3 Wednesday May 23rd 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (POSTPONED)”

What? No complaints yet? No “NightmareHost” comments?

The trolls are slacking.

Everything is being deleted.

Having outages intermittently for a few weeks now…no explanations, no acknowledgments. Running slow, timing out etc.

I’m having troubles with ALL my sites under Dreamhost ¿Could someone help me? the sites running Slow and are not showing anything

All of my websites are slow too – have been since the beginning of this year. Page loads of 10, 15 or even 45 seconds are not uncommon. Most forums refresh pages in less than a second. DH takes many, many seconds to query the MySQL server and complete the page. That hurts traffic flow since most users don’t like to wait, they will bounce from the site. Plus, ads are paid per thousand impressions, if the pages load slowly, ad impressions go WAY down and your income suffers.

Old hardware + 12 year old network hardware + too many websites on shared hosts (over selling a server) = sssllloow page loads.

Is Kraz server on this network?

“resulting in all web hosting services in that data center being offline” how do we know if we are concerned ?
My server is called ‘agba’
thank you

I haven’t had any downtime at all since March (at least not according to my uptime monitor which checks every 5 minutes, 24/7). Maybe other people’s issues are server related. It seems like DH is really improving the network hardware. Maybe the days of network-wide outages are over?

FPT ain’t connectin’…. Y’all know how damn inconvenient this is??? Gotta get a video to some clients in Caracas, VE. A lil’ over $25,000 on the line…

I think I’m being pretty laid-back given the stakes. Been with ya since ’98, about 13 years. Never had this much trouble. WTF???
Get it together, or get the hell outta my business plan.

Y’all have a good day..
Kevin Bohn
Shopdawg.com

Again and again and again.
Downtime, slow, no response!
I need to work but dreamhost does not!

May 23rd, 2012 at 6:48 amTime to change Says:

Dreamhost, you really suck. Your Planned Downtime is awesome because you dont count it in 99.99% Uptime.
Time to change. seriously

Sites are down again, with nothing posted here that is actually relevant to the problem.

My site is completely down now, after running extremely slow all morning. Please post an update.

@Nancy: Please contact support with examples and we can take a look.

@Brian D: Please contact support with examples. We are not using old hardware or 12 year old networking hardware.

@Ron: No, kraz is in our lax datacenter. The maintenance is for tonight, hasn’t happened yet and hasn’t caused any downtime yet.

@Julien: Your webserver is in our irvine datacenter, and will not be affected.

@Ben: We really are investing a lot in network upgrades and stability, and we do expect much fewer issues in the future when all upgrades are complete, and even improved stability now with partial upgrades done.

@Shopdawg: This really isn’t the best medium to communicate to us about an urgent immediate issue not related to any downtime posts here. I see one of our support people wrote you after this post, but we haven’t received a response from you. Your webserver has a 230 day uptime and no reported ftp issues.

@TImeout: Please contact support with details, there was no information provided in your post to assist you further.

@Time to change: Please contact support with details, there was no information provided in your post to assist you further.

@Kate: Please contact support with details, there was no information provided in your post to assist you further.

Aaaaaaaaaaaand my site is crawling slower than toothpaste.

iheartpapers.com
veryusartists.com

I’ve already tweeted and submitted a support tix which got moved to “severe problem queue. This queue is for major outages that cause a large amount of support to come in.”

@Rachel @ Grasping for Objectivity: Please take a look now, I’ll reply to your ticket about your issue.

@Tam Nguyen Photography: I couldn’t detect any slowness loading your site at this time. Please submit a ticket with specific examples of slow pages and I would be happy to check it out for you.

@Kate: The issue that was affecting your sites, is now resolved. You will be getting a reply to your ticket shortly.

I have been caving my programmers skull in because our site is soooooooo slow. One of my outsourced programmers [no vested interest] tells me it is NOT your site it is Dreamhost. What is going on?

guys my webmail is crawling. intermittently several hours delay between send and delivery for the past couple of days.

Hi,

My web site is still ok. Not receving/experiencing slow connections….

Maintenance tonight has been postponed. No service interruption has occurred.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Delay in sending email from web servers (RESOLVED)
11:17 am

Delay in sending email from web servers (RESOLVED)

Posted (May 22nd, 2012 at 11:17 am PST) by andrea

The mail relay machines for your web servers got backed up earlier today. We are working on processing that mail as fast as we can, but it will likely take another few hours to clear that mail queue. In the meantime, you will see delay in sending your php- or sendmail-based email or those created by your cronjobs. No mail is lost, we’re processing it all in the order they are generated on your server. We apologize for the delay and the inconvenience this is causing.

Update May 22nd 10:04pm PDT We’re happy to report the mail server has caught up now. Again, we are very sorry about the inconvenience this has caused. -Dan J

Update May 22nd 8:37pm PDT The mail is still being processed through the central relay server. Unfortunately it’s taken longer than expected but we are making progress. Again, we are very sorry about the inconvenience this has caused. -Dan J

Update May 22nd 3:53pm PDT We’re continuing to clear the mail queue on the relay server. To reduce the load, we are only processing a certain amount of queued up mail at a time. We apologize for any inconvenience this is causing. – Andrea

Update May 22nd 1:05pm PDT The mail queues have been catching up as expected, but we’ll keep this post open until we are back to normal. Thank you for your patience. – Andrea

This entry was posted on Tuesday, May 22nd, 2012 at 11:17 am and is filed under Email Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

13 Responses to “Delay in sending email from web servers (RESOLVED)”

May 22nd, 2012 at 11:28 amjoe webserveruser Says:

thank you for posting this! the problem has been happening since at least 5pm PDT yesterday. why does it take so long for you to notice these things?

I’m assuming this does not at all effect Gmail hosted mail? My outgoing mail has just stopped sending, so I wanted to make sure these aren’t related before trying to track down Google’s tech support.

It does not appear that mail is being processed in the order in which it was generated. Are you sure about that?

no updates for 3 hours. mail still not being delivered. what’s up?

Why on earth are you “reducing the load” on the mail server? That’s flat-out stupid. You mail server should be running with maximum load to deliver the backed up mail as quickly as possible.

Updates like this make it sound like you have not clue at all about running a mail server. Which could explain why there are so many problems with your mail servers so often.

Update? Mail is still down.

Can we have an estimation of the end of that issue ?
I mean if you see that you’re sending every hours 5% of the queue soon 20 hours its done ?
Or can you tell us, you did more than 50% (or not) ?

Can you please explain Andrea’s 3:53pm comment? It makes no sense.
Can you also provide an ETA for return of service? If you can not provide and ETA, can you explain what this issue is which prevents you from providing an ETA?

If you cannot provide an explanation of why you cannot provide an ETA, can you please have someone who can provide an ETA update this status report?

All in all, you have not provided ANY information at all here which was not already completely obvious to all affected parties. You’re supposed to be better than that.

According to your most recent update to this issue “the mail server has caught up now.” However, we have many messages which have still not been delivered. Following is a sample listing of missing messages.

messages sent to michael@ at:
5/21 4 PM, 5 PM, 6 PM, 8 PM, 9 PM, 10 PM, 11 PM
5/22 12 AM, 1 AM, 2 AM, 3 AM, 4 AM, 6 AM, 7 AM, 8 AM, 9 AM, 10 AM, 11 AM, 12 PM, 1PM, 3 PM, 7 PM, 8 PM, 10 PM

messages sent to scheduling@ at:
5/21 4 PM, 5 PM, 7 PM, 8 PM, 9 PM, 10 PM, 11 PM
5/22 12 AM, 1 AM, 2 AM, 3 AM, 4 AM, 5 AM, 6 AM, 7 AM, 8 AM, 9 AM, 10 AM, 11 AM, 12 PM, 3PM, 7 PM, 8 PM, 9 PM, 10 PM

messages sent to reminder@ at:
5/21 4 PM, 5 PM, 8 PM, 9 PM, 10 PM, 11 PM
5/22 12 AM, 1 AM, 2 AM, 3 AM, 4 AM, 5 AM, 6 AM, 7 AM, 8 AM, 9 AM, 10 AM, 11 AM, 12 PM, 3 PM, 5 PM, 6 PM, 7 PM, 8 PM, 9 PM, 10 PM

messages sent to announcement@ at:
5/21 4 PM, 5 PM, 8 PM, 9 PM, 10 PM, 11 PM
5/22 12 AM, 1 AM, 3 AM, 4 AM, 5 AM, 6 AM, 7 AM, 9 AM, 11 AM, 12 PM, 3 PM, 6PM, 7 PM, 8 PM, 9 PM, 10 PM

I hope you will act quickly to my support request (submitted through official channels) to save and check your logs, and investigate. The last time this occurred I was told that you couldn’t investigate because the logs had already been deleted (because it took several days for you to begin investigating).

We still continue to see messages not delivered as follow:
mails sending from webform module of Drupal sent today (23th may) at 10:53, 10:55 and 11:26 Italy local time.

We hope you investigating why still continue e-mail problems

This has happened at least 3 times so far this year alone. The “queued” messages don’t go through . If you say these messages will be delayed that is fine, but they are not delayed. They are gone for good. I have not received automatically generated e-mails from yesterday.

still unreliable.

why is this marked “resolved”??? mail is still down. does this mean that you have given up trying to fix it?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Immediate Maintenance Required on Discussion List Servers ( RESOLVED )
12:56 pm

Immediate Maintenance Required on Discussion List Servers ( RESOLVED )

Posted (May 16th, 2012 at 12:56 pm PST) by Oscar

Our system administration is performing maintenance on our discussion list servers. We expect all discussion list services to be down approximately one hour. We apologize for the inconvenience but it was deemed necessary as we’re setting up an improved (and more reliable) backup system for the lists.

UPDATE May 16th 3:18PM PDT The maintenance has completed. You are archives are now accessible. Again, we apologize for any inconvenience this has caused. If you notice any problems, please contact support for assistance.

UPDATE May 16th 2:49PM PDT The maintenance is taking longer than expected. Your list archives will remain unavailable until the process completes. We sincerely apologize for any inconvenience this is causing. Please check back for updates.

This entry was posted on Wednesday, May 16th, 2012 at 12:56 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking issue at the Irvine datacenter (RESOLVED)
12:38 am

Networking issue at the Irvine datacenter (RESOLVED)

Posted (May 16th, 2012 at 12:38 am PST) by andrea

We’re experiencing connectivity issues at our Irvine datacenter. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update May 16th 1:39am PDT Tonight’s connectivity issues appeared to be a result of a drop out experienced by one of our transit providers. They are dispatching an investigation team to help assist us in our diagnosing the exact issue that occurred. We are not expecting any further problems, however we will continue to monitor the situation, as well as assist our transit provider on finding further details on this issue. We’re resolving this case for now, but if you have further issues, please contact support for assistance.

Update May 16th 12:47am PDT We are happy to report that service has been restored. There was a slight outage that recovered and we have our admin on site making sure no further incidents occur. We are currently diagnosing the cause of the connectivity issues. We apologize for the inconvenience.

This entry was posted on Wednesday, May 16th, 2012 at 12:38 am and is filed under General Outages, Major Outage. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

39 Responses to “Networking issue at the Irvine datacenter (RESOLVED)”

At last. Thanks for acknowledging there is an issue guys. Sites have been down for 1 hour now.

again, my site is down.

Waiting for a long time for more than 2 hrs

not good, the biggest day of traffic flow for me on 3 of my sites :(

anybody can recommend another reliable webhost. Dreamhost has been really bad lately.

nice… all our marketing efforts down the drain with all our money… This is definitely bad news for an auction site..

What servers are affected?

Ana: as far as I know *all* dedicated servers are located at Irvine :(

I don’t know what the fark I can say about Dreamhost this year…. I want to be out of Irvine datacenter or I should leave Dreamhost…

It has to be said guys I want you to move me from the bloody Irvine Centre.

It is rubbish!

You guys make bold promises but NEVER deliver!

Andy

Seriously bad form guys… Three outages in two weeks. I’ve never had a host which has caused the amount of downtime I’ve had with dreamhost in these last two weeks in all the years of hosting I’ve used.

Please fix it immediately. Mine around 8 websites are down.

It is up FINALLY…. Thanks for support guys for your efforts to fix the issue immediately

Mine is still down, been down for about 1 hour+

This is just staggering. It’s like the film Groundhog Day – the same nonsense over and over again. 100% uptime, 100% uptime!!!

Good news! My email worked during this crisis but then again, my email is on Google server… Ohhhhhh, maybe dreamhost should consider selling to google? lol sorry had to make a funny…

Still down here. Was there a planned maintenance going on at the same time? I seem to remember a post on that that disappeared.

this is stopping me from registering a new email adress on one of my domains to use for battle.net which means its stopping me from DIABLO 3. :(

It’s official – the Irvine Data Center is haunted! Film at 11!

The Irvine data center is where old, tired, worn out Compaq 486 servers go to live out the rest of their useful life, serving HTML code and MySQL databases from 5.25″ full height, 3.6GB Micropolis 50 pin SCSI hard drives over a Thinnet 10base2 coax network ring.

huh ! still down for me half an hour after last update !

still down for me too, cannot ssh into vps or access hosted site.

Mine still down. dreamhost.com not loading either though… hmm..

makes one wonder where they host dreamhoststatus.com…..maybe THAT HOST is REALLY reliable????

And it just came back again….let’s see how long until the next outage…

@MAL we should take bets lol some of us could win the money back we are paying for downtime on >_<

My sites are still down, for at least 4 hours now.

The one thing I’ve noticed is these threads get far fewer responses now, is it because people are used to it or did everyone with critical sites move away alread (like I should have done).

I think dreamhoststatus.com is hosted at linode.com…

May 16th, 2012 at 6:49 amBrian Richardson Says:

All of my sites are still down, so there is still some unresolved problem.

We’re back up after around 8 hours of downtime and a massive hole in the stats.

Thanks for fixing the issue.

http://www.onelookitsolutions.com/

Four months in and i’m getting a bit worried. Is this the norm?? Happens too often. Way more than my previous host. The service is unreliable. Thankfully, my live sites are not being hosted here. Doubt they ever will be if this is the accepted service level. 8 hours and counting.

hi there..

my site is down..

any ideas?

my sites is still down,

This is deplorable, the last time that I have my site in Dreamhost

whats going on with the VPS and MYSQL?

down like a 100 yr old mans erection!

I keep suscribed to this feed to be glad i droped dreamhost after being a client and suffering it for 2 months.

@Cesar, I know what you mean. I also cancelled my VPS account with Dreamhost a week ago and I’m so glad I did. Good luck Dreamhost, I sincerely wish you will get your act together, for the sake of your customers.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Transit issue between LAX and Alchemy datacenters (RESOLVED)
12:19 am

Transit issue between LAX and Alchemy datacenters (RESOLVED)

Posted (May 12th, 2012 at 12:19 am PST) by Brian@dreamhost

Currently, at this time we’re seeing network degradation between our LAX and Alchemy datacenters due to maintenance by our transit provider. We are investigating this issue and will have more info as it progresses. This issue should only affect a small amount of customers with mysql databases that are in opposite datacenters, or users using webmail, on our homiemail-sub5 mail cluster — Brian H

Update May 12th 2:10am PDT This issue should now be taken care of at this time. We were able to configure a different link with another provider, and restore service. We apologize for any issues this may have caused.

Update May 12th 1:31am PDT We’re still seeing problems with inter-datacenter communications and are working with our network provider to get this resolved. An update as to what actually happened; we were aware of this planned maintenance, and didn’t expect it to be a customer facing issue. We apologize for this, and will announce all carrier planned maintenance windows in the future.

This entry was posted on Saturday, May 12th, 2012 at 12:19 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

36 Responses to “Transit issue between LAX and Alchemy datacenters (RESOLVED)”

homiemail-sub5 mail is my cancer!!
no email service. No pop3 no IMAP.
email service is down (again and again)

againnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

And where is the redundancy? maintenance on one transit provider should kick in the backup?

“This issue should only affect a small amount of customers”

I feel so special, like I’m part of an exclusive club!

I was going to submit a support request – thanks, at least, for putting the info out there. I hope it’s resolved soon!

I’m grateful for the comments. At LEAST I know that I’m not alone in having mail problems tonight.

All my sights are down as well.

Databeses are all down for our sites.
“Error establishing a database connection”
Our server is aldebaran.

Please fix this issue…

OK, It’s been over half an hour. Let’s get this resolved already…

The trouble started around 11:30pm.

We are one of the sites affected. Please let us know when we can expect our database-dependent sites to be functioning again.

What about redundancy? We received a nice letter last week explaining all about how Dreamhost has fixed all of the problems from January and February with a lot of technical explanations about added reliability. I guess we have another similar letter of apology to look forward to after this?

ps50184 DB’s are down too.

I guess that explains why I keep getting a “INTERNAL ERROR CREATING DB: connect_admin failed for chimaera:wrobel. Please let support know!” when I try to create a new MySQL DB.

helloooooooooooooooo? going on two hours now…… againnnnnnnnn…

I am using Google Apps, and it doesn’t work too. Emails are not sent through website with SMTP. It gives this error:

unable to connect to ssl://smtp.gmail.com:465 (php_network_getaddresses: getaddrinfo failed: Name or service not known) in /*****/libraries/phpmailer/smtp.php on line 122

My problem has something to do with this situation, does anyone please confirm that?

looks like http://discussion.dreamhost.com/ has its db and server in different data centres as well:

502 Bad Gateway

nginx/0.8.53

This seems to be a once-per-month-or-higher occurrence — another several hours of downtime for many of my clients’ websites. All this after the recent apology letter from DreamHost’s CEO and promise that it would never happen again — I’m starting to wonder who else might appreciate my VPS money a bit more. I’m open to suggestions.

…and why are my web server and database server in opposite datacenters anyway? Who designed this network?

Hi!

We are not using MySQL whatever but all our sites are down.

http://morsm.al

I’m so sick of this.

And miracuöously I’m always one of the “small amount of customers” who are affected.

“Comments posted here may not be viewed by DreamHost staff at all.” Thought so.

homiemail-sub5 not delivering POP mail yet again. Oh, how I hate thee, homiemail-sub5.

May 12th, 2012 at 1:45 amPatrick Rosenheim Says:

again? seriously? fucking AGAIN?

mail is down

A-fucking-GAIN!

I demand yet another DH Apology letter, FFS!

Seriously, this is now well beyond being ridiculous

my sites are back online. 10x.

I was just able to create a new database without getting the error.

My website is finally back online after more that 3 hours. It doesn’t have to be this way.

my site is offline cant even get into my panel to contact support, on the other hand the webmail seems tobe working fine..what gives.

I’ve gotten at least 20 mother effing emails regarding my site’s status change. UP and DOWN. LEFT and RIGHT.

The title of this post should NOT include the word resolved. Gah.

May 12th, 2012 at 8:58 amRicardo Daniel Says:

Error establishing a database connection

Getting the following http error for both my sites:

————————————————–
Forbidden

You don’t have permission to access / on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
——————————————

Anybody else experiencing this? Anybody know whether Dreamhost is doing anything? Status messages indicate that there are no longer any problems.

thx. – tom

Hello i am trying to login to my email accounts and i keep getting the error message.

” ERROR:
ERROR: Connection dropped by IMAP server.
Query: SELECT “INBOX”

ERROR:
ERROR: Connection dropped by IMAP server.
Query: SUBSCRIBE “INBOX.Sent”

Both of my websites AND my admin are down! I am getting this error message

Notice: Error: Table ‘opencart_store1.weight_class’ doesn’t exist
Error No: 1146
SELECT * FROM weight_class wc LEFT JOIN weight_class_description wcd ON (wc.weight_class_id = wcd.weight_class_id) WHERE wcd.language_id = ‘1’ in /home/maril80/marilyntov.com/system/database/mysql.php on line 49

Anyone heard back from dreamhost today?

WTF is this? Mysql is down since more than 12 Hours and no updates here after so many hours..really embarrassing

I am so happy I am one of the lucky ones. Please fix this soon. Unprofessional to recommend a site that is broken.

We were just picking up momentum with our marketing and now everything’s going in the bin including our money! When is this problem going to get fixed??

We’re freaking running a penny auction site! if this goes on any longer we’re going to lose serious money in the thousands!! Please Dreamhost… im begging you! Fix the problem!!

Which transit provider? AboveNet? Comcast?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)
1:30 pm

Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)

Posted (May 10th, 2012 at 1:30 pm PST) by Ralph

This is the second step in our DreamHost Network Improvement Series.

On Monday, we will be upgrading the Border Hardware at our Irvine Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4-hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, so your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 14th 10:08pm PDT: We started our maintenance at 10pm PDT, and experienced less than a minute of downtime. Our customer test domains are loading and are stable, and we are confident that this case can now considered to be resolved. Our Irvine datacenter now has new border hardware.

If you’re having issues with your sites, it’s likely unrelated, so please contact support with the details, so we can assist you.

Update May 14th 12:23pm PDT: We are going ahead with this maintenance tonight, May 14th, starting 10pm PDT. We will update this post with any status changes and progress reports.

Update May 10th 11:20pm PDT: We will need to reschedule this maintenance for another window unfortunately. The changes were successful however we were not comfortable with the state of the physical set up so we reverted back to a stable state. We will be creating a new status post once we find the best time to reschedule this event. Apologies we were unable to finalize this upgrade.

Update May 10th 11:10pm PDT: We have rolled back the changes. There were complications with the new hardware install. More information to come shortly. The sites we’re using to test are now back up, but on the old hardware.

Update May 10th 10:50pm PDT: Still working on getting everything working correctly on the new hardware. Service is still affected at this time.

Update May 10th 10:35pm PDT: Cut over is starting now. Service will be interrupted as we switch out the old hardware with the new. Please stand by. Your patience is greatly appreciated.

Update May 10th 10:20pm PDT: Final checks are being run and then the cut over will happen shortly. Next update should announce that the cut over to the new hardware has begun.

Update May 10th 10:10pm PDT: Finalizing router configurations before we start actually cutting over. Once we start cutting over, service WILL be down. We expect downtime itself to be minimized as much as possible.

Update May 10th 9:50pm PDT: Our maintenance team is coming together and getting ready to work. All important updates will be provided as soon as I have them. Thank you for your patience.

Update May 10th 3:10pm PDT: The maintenance on our border routers will begin at 10pm PDT. We expect the maintenance to complete by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through 2am PDT. Here is our expected sequence of events during the maintenance:

– Network services in our Irvine data center will be taken offline between [10-10:30pm PDT], resulting in all web hosting services in that data center being offline.

– A new border router that has been set up and prepared for operation will be connected to the network and tested, with an expected duration of [30 minutes] for the testing to be completed.

– Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Irvine and all web hosting services will be back online; we expect this to occur by [11pm PDT], however it is possible that network services will not be restored until [12 midnight PDT].

– In the event that the new router does not pass all tests by [11.30pm PDT], we will restore services on the current network router hardware, so that in any scenario, all network services are restored by [12 midnight PDT].

We will continually provide updates through the maintenance, especially when the outage begins and when we feel everything has returned to normal.

This entry was posted on Thursday, May 10th, 2012 at 1:30 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

44 Responses to “Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)”

May 6th, 2012 at 9:45 amDan Poynor Says:

There are other outages not listed on dreamhoststatus.com. Here’s the message that appears in my panel…

Current Outage: Hardware replacement on your VPS hosting server homie-vserver192 (updated 1 day 9 hours ago)
Hello,
Due to issues with the RAID controller on the VPS host server ‘homie-vserver192′, which hosts your VPS machine, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.
We apologize for the inconvenience caused by this unplanned outage. We will keep the announcement which appears in your Web Panel updated with as much detail as we can provide. If you have any questions or concerns, please contact us back under Support > Contact Support, in your panel.
Please note, this is regarding the VPS host server ‘homie-vserver192′ and no other services (email, MySQL) or servers are affected.
Thank you for your patience!
The Happy DreamHost Server Fixing Team

Dan Poynor, we both share this problem. This looks insane… Current Outage: Hardware replacement on your VPS hosting server homie-vserver192 (updated 1 day 6 hours ago). I need any information but there is no answer :-(

Our website is currently down and has been for the last 3 hours. When is this going to be resolved? Thanks.

I literally finished my first proper website which I have with Dreamhost today and started to publish and share it across multiple blogging networks and two hours into my marketing push the website goes down.

What is going on? It’s really not acceptable at such a crucial time after submitting it so that people can take a look at it.

RE: Current Outage: Hardware replacement on your VPS hosting server homie-vserver192

I have the same issue/problem/NIGHTMARE!!!
Earlier today I was able to FTP the site, and all files were from a few weeks ago. It appeared DREAMHOST recovered a backup that was maybe 10 to 15 days old. I would have been able to live with that, I guess. My database of 2.4GB was still current it appeared.

BUT NOW, when I FTP the site, it’s COMPLETELY empty! ALL GONE!!

ROOT folder shows only .cshrc, .bashrc, alias, Maildir, and log (both folders are empty)

ALL SITES, all files have disappeared!!!

May 7th, 2012 at 4:37 amdreamhostishell Says:

They did the same to me. And from what I heard, it seems to all be lost. I was told they could credit but could do nothing for compensating for material. Not sure if that means everything was lost. Better not be.

May 7th, 2012 at 6:04 amMikey Pinger Says:

Boom goes the dynamite.

May 7th, 2012 at 7:47 amShawn Gossman Says:

Good job for announcing this several days in advance. I will make sure my users know about it.

A full working day without access to websites and information listed the sites is more than 3 weeks. I had to take a day to forget about customer complaints. Do you know when will the whole process over? I’m praying than several days in the late fall of Virtual Private Server.

This site is a sham. There has been an ongoing server outage on server ‘kuwaitcity’ for over three days now and no mention of it on this site.

The current message is below, but this has been going on for days now.
“Current Outage: DreamHost server Kuwaitcity being moved to new hardware (updated 22 hours 19 mins ago)
Hello,
Your webserver ‘kuwaitcity’ is now running off of the new hardware and your sites are being restored at this time. We’re seeing some intermittent load spikes on the server that are causing sites to become unresponsive, but we are working to address those spikes as soon as they come up.
While we still don’t have an ETA on when this restore will be complete at this time, we’ve tested several sites that were reported down within the last couple of hours and verified that they are online and working at this time.
We will keep you updated with as much detail as we can provide. If you have any questions or concerns, please contact Support and we’ll be happy to help you further.
Thanks!

DreamHost Support Team + support@dreamhost.com

May 7th, 2012 at 5:18 pmAmerican Ex pat in NZ Says:

@ Jonathan: We’ve still got 26 sites down (up from the 30 odd that it was) ~ this has been ongoing since our Wed 2 May 2012 on the ‘kuwaitcity’ server. Sadly this outage and lack of communication re: fixing / restoring on Dreamhost’s part may very well sink my husband’s business. :(

“dreamhostishell Says:

They did the same to me. And from what I heard, it seems to all be lost. ”

At least my sites are ALL back up online, with most of it recovered. The database went from something like 1.4GB to 800GB, but it was still a recent version with what appears to be only some attachments lost.

relatively happy now…

May 7th, 2012 at 7:26 pmit's getting better! (grin) Says:

You folks don’t understand that “this is the second step in our DreamHost Network Improvement Series.” Be patient!! They still have 1,375 steps to go, and everything will work (for about five minutes).

I have been waiting for over 4 days now some answer on support.
DH is getting terrible. Stop partying and work!

I’m calling BS on a failed RAID card and moving to new hardware. I get this excuse every time there is downtime, including as recently as March. http://www.dreamhoststatus.com/2012/02/27/web-vps-host-server-homie-vserver19-moving-to-new-hardware-and-restoring-from-backups/

C’mon guys, I don’t see a reason that it would take 3 days to “move to new hardware”. A simple human response to my tickets would go a long way. I’ve been with you since 2007 and never had a ticket go unanswered for more than 6 hours.

“Hello,
You are receiving this message because your web hosting services are being affected by hardware maintenance due to issues with your VPS web hosting server homie-vserver19.
Due to issues with the RAID controller, we are in the process of replacing the hardware for this server and we are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow your hosting services to remain online for the duration of the process.
The server is currently still restoring, having around half of the VPS restored. All guest machines and web services are up. You may notice intermittent slowness as a result of the restore process. Our admins will continue to keep an eye on this and make any adjustments necessary to make sure everything is running as smoothly as possible.
We apologize for the inconvenience caused by this unplanned maintenance. If you have any questions or concerns, please contact Support under Support / Contact Support, in your Web Panel. This notice will also appear on that page in your Web Panel and will be updated throughout the process with the latest details we can provide.
Thank you for your patience!”

May 8th, 2012 at 1:45 pmAnastasia L Says:

Server microscopium on day 11 of issues, been down more often than up. I am now getting emails from support talking about big improvements coming soon, but still no word on when I can expect my websites to function normally.

Oh the irony of Dreamhost sending out e-mails explaining how things are going to get better when our VPS PS60210 has been dead in the water since yesterday afternoon and our support tickets generate no response. Until they learn what good customer support is, all the hardware upgrades in the world are pointless.

May 9th, 2012 at 9:22 pmSeriously? Says:

All my sites are down.

here we gooooo AGAIN!!!!!

all my sites are also down right now :-(

is the server “acrab.dreamhost.com” down???

Today I lost 4 customers because they are tired of falling from their webs. After 4 days, all the websites are without service. Are sites with more than 4 weeks old and customers are driving me crazy. A few months after my VPS have fallen for 6 days (How do you explain to a customer that your business depends on its website that will be 6 days inoperative?) Lost 6 clients and two of them very important to me. Despite paying a substantial fee and accommodation have 3 different professional not get any answer. Dreamhost has shown that it is only for individuals. If you are using the server for something professional, you can sink and ruin. This year the crisis and I’m losing customers due to falls from Dreamhost. I have always been faithful to Dreamhost but I can not let me spoil slowly.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

BRACE YOURSELVES, UPDATE IS COMING.

WHAT THE HELL IS GOING ON?! I never type in caps, but this is just ridiculous! 100+ sites are on/off on/off, intermediate connectivity and then nothing. Entire staff is answering calls/emails only because of this! DreamHost, don’t become NightMareHost!

All four of my sites are just not loading (howmotherhoodchangesus.com; wiredforstories.com; documenting medicine.com; and oakwoodproject.org)

What is going on and when will these be fixed? I had a big event today and fortunately had downloaded a static version to show off my laptop because of the problems I’ve been having with Dreamhost. I hope this gets fixed or will have to recommend my students at Duke build their sites elsewhere.

Ah my website is down! (www.photoshoptroll.com)
I’m losing a bunch of traffic, I hope it comes back up soon. :(

Site keeps going up and down…we just put out some great content and now nobody can see it. It’s killing our traffic!

Fix this asap, I’m getting sick of these server issues…and we are about to switch hosts.

down down down down down down down down down down down down down down down down down down down down

Please the website say me

Error establishing a database connection

Is about the maintenance ???

which machine is in which data center? Thanks.

@LakeRat Unfortunately we do not have a real good way to list all affected machines. We are in discussion now on how to make this information more accessible moving forward. We will likely have an area of the web panel where you can check on this information. Sorry that this information is not available at this time.

May 10th, 2012 at 10:45 pmShawn Gossman Says:

ds2032 is down, I assume its for these updates which is fine seeing as I warned everyone :) If it shouldn’t be down, well then here is my report :p

@Shawn You definitely should be down right now. Hang in there!

Pine is down… Good job guys… This is like the 20th time server goes down since April…. Anyone remember Dreamhost’s uptime guarantee? 9%?

ps95872 is down and i experienced 1 hours of downtime yesterday too. Am I affected by this maintenance work?

@Jia your private server is at Irvine so you are affected by this outage. I’m not aware of any Irvine issues yesterday and we were watching it all closely so I can’t answer to that unfortunately.

@Pine – They have a guarantee? I’ve been with them for several years and within the past year they’ve gone from a company I’ve kept using to a company I’m rapidly planning on ditching the first chance I get. Non-stop problems and constant outages are just getting old.

I’m hosting a few production website for our clients and in the last two days I’ve experiences two faults with the server. Should I be moving my hosting to another provider if this situation will continue? Maybe media temple instead?

Please provide an ETA to when I can expect to have my services back up and running. Reading some of the comments above it seems data was lost as a result of this update, am I at risk of loosing this data as well?

Definitely understand your frustration. This is why we are performing these series of network improvements. DreamHost is growing up and we are unfortunately hitting major growing pains. The good news is we do have the minds in place to survive. I empathize with your issues. It’s difficult to be a customer of a company like ours with stability issues and it is hard on the employees as well. We will continue to work hard for you guys and make the decisions we feel will make us the best hosting company around. We have listened to you and we do feel we are failing you and these improvements is us doing our best to make this up and give you back your trust in DreamHost.

Also, reading through the network status page, it appears a lot of issues have been plaguing the Irvine data center. Can I request that my private server be moved to another data center without these issues?

Seems like all my websites are affected? Everything is unreachable at the moment. I’ve submitted a ticket regarding this but have not heard anything back. I have clients needing to get in touch with me. Lucky I still have the facebook option, but just seems very unprofessional DH!

@Jia All of our data centers are getting their networks updated. We did just roll this one back, but Irvine has has upgraded hardware since the last incident and has been working fine. We are actually putting a beefier system in place at Irvine. You can also submit a support ticket and put Attn. Ralph and I can investigate your request further.

@Ralph – Thanks for that. I can see the site is now back up and running. I appreciate your timely and prompt response to this issue. I could like in the future to be made away of these types of maintenance work which might impact my services as I do host these for our clients properties hence why the VPS solution.

I will put a new support ticket in for your attention so we can ensure the services won’t be disrupted without prior warning.

Cheers!
Jia

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)
9:00 pm

Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)

Posted (May 9th, 2012 at 9:00 pm PST) by Ralph

This is the first step in our DreamHost Network Improvement Series.

On Wednesday we will be upgrading the Border Hardware at our LAX Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4 hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 10th 1:10am PDT: We are considering this maintenance resolved. Thank you for your patience as we worked through this first installment of our Network Improvement Series. We are excited about these changes and what they mean to our service and what they’ll mean to you, our customers, as well.

Update May 9th 12:20pm PDT: Everything remains stable from what we are seeing. We are still working on getting our backup provider up and working correctly. Once that happens we will give everything the all clear.

Update May 9th 11:45pm PDT: All test sites are working correctly again. We are nearing the point of calling this complete however we are still verifying that all is clear. So far everything has gone off without any major surprises.

Update May 9th 11:30pm PDT: We are nearing the end of the maintenance. We are already seeing service being restored to some sites we are checking. Thank you again for your continued patience. We hope to have the maintenance window closed well before the full 4 hours. We will continue to keep you updated.

Update May 9th 11:10pm PDT: We are in the process of switching over to the new hardware. Service interruption will be happening at this time and we will do our best to limit the downtime as much as possible. Thank you for your patience.

Update May 9th 10:50pm PDT: We are currently switching over to the new protocol that our new juniper hardware requires. We expect a minor blip at this point of the upgrade.

Update May 9th 10:20pm PDT: We are finalizing router configurations from the old hardware to the new. Once that is done the actual cut over to the new border will take place.

Update May 9th 10:10pm PDT: We are still finalizing some preparations. The actual part of the maintenance that will cause service interruption has not started yet. We will keep you updated.

Update May 9th 9:10pm PDT: We are getting ready for the maintenance to begin. We still expect the start time to be 10pm PDT. Just letting you know we are here and ready to do this.

This entry was posted on Wednesday, May 9th, 2012 at 9:00 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

80 Responses to “Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)”

Is this why I can’t upload anything to my FTP?

jump the gun? It’s only 9:25 pdt and my sites are done. :P

Please hurry – mine went dark at 1:04CST. How long should sites be down?

“Minor blip” eh? I’m at 8+ min and counting…

this is really annoying, i was in the middle of developing… when my php files were uploading.

All my sites are down and I can’t connect through ssh neither… I expect that this happens fast

all sites down,my god!

I wish there was an estimate for how long our sites will be down. I was in the middle of an update, and I’d like to know whether to wait it out or give up and try again later.

This is so unacceptable. what the &^%*&^, time to drop …. just kidding. Thanks for making the updates and sorry you guys deal with all the complaining.

Come on people. For those with nothing better to do that complain…this is a huge job. They allotted 4 hours. Although it most likely won’t take that long…they’ve set expectations. You’re “minor blip” was for only one particular step in the upgrade process. 8 minutes is definitely not enough time to get this done. Hold your horses & chill out.

May the force be with you, DH Systems & Network Engineers.

Anyone know if this would affect mail? Our site is up but mail is down. Webmail interface works but no mail going into it.

Coyote, that is the problem with your company. There should be 0 downtime.

this is another nail in the coffin.

dear friends, respect from Turkey… how long will take it? from now for 4 hours?? hope ı m wrong…

Dammit! Just my luck…. I took the day off work to develop my site and wouldn’t you know it, its down lol!! Keep up the good work DH!!!

Hey guys (and to all who want to complain).

Dreamhost announced this maintenance several days ago, so you had plenty of time to prepare. If you didn’t know about it just subscribe to their twitter channel.
As for the how long these interruptions will last: As the title says the maintenance starts at 10pm PST and the maintenance window is 4 hours. This means you should expect to see service interruptions up until 2am PST.

Just stay cool.

IF the highway is broken, constructors close the road and fix it at 4am in the morning, to avoid any possible traffic chaos. Right? and Dreamhost, please tell me why the hell are you doing this at 10PM? It is the rush hours. Why don’t you do it 2-6 AM (PDT) in morning?

Phil…My company? What do you know about “my company?” BTW…anyone that promises 0 downtime is a fool. 3 nines isn’t even all that realistic. Especially without redundancy. I’m sure your shared hosting account or even your private server has no redundancy built into it. Good luck getting that without actually hosting it yourself (or without spending a boat load of money). Network upgrades are some of the most difficult to perform with minimal downtime. As a Systems Engineer myself, I can tell you that a 4 hour window is MORE than reasonable. Again…it likely won’t take that long, but they’ve set those expectations because of entitled customers like yourself that know very little about actually running data centers like these.

please hurry

Finally, my site is up again. Thanks DH.

@mustafa

Dreamhost host websites from all over the planet.
It doesn’t matter when they do the maintenance. It’s always rush-hour somewhere in the world.

Why should I have to follow them on Twitter to find out? Especially when there’s no way to know who’s in this data center until the sites go down. Would it be so hard to send an email to everyone who will be affected? Lord knows I get 9,000 emails I don’t want from DH over domain renewals. It’s not a big deal for my sites to be down for a while, but I’d rather it were handled in a somewhat professional manner.

Great job DH!!! Thanks, my site is back up and running!! :) :) :) :)

Sorry but weekends may be better in the 2-6am window…for us in the rest of the world eg Australia this has occured in the middle of a Thursday…email and sites all down …anything over 1/2 hour is poor.
Why do I have to subscribe to get updates…don’t DH trust there own email?

Well said Coyote! Agree 100%

Looky there. My site is back. Four hour maintenance window nearing completion in half the time. And actual downtime was only about an hour. How’s that for meeting expectations?

@Tim

You have a point.
The web-panel should have info about which datacenter we’re in.

However email is a stone-age and rather inefficient form of communication for these kind of announcements. Everyone who’s interested in keeping up-to-date with maintenance info should subscribe to their twitter channel anyway. They always announce their plans there without fail.
Why should they duplicate their messages?

*thumbs up* Coyote … also 100% agree with earlier post!

Now then, is anyone else not able to access the DH panel? Or is it just me?

Though I was anticipating this, it kinda caught me off guard a little while in the middle of working on my site (it’s 3am EST now). I should really be getting some sleep. DH says things are apparently working now, but I’m confused due to the fact that the panel is still offline as well as my sites…

Which sites are they testing, the panel is kind of a big deal.

@Jon

panel is working for me, what is your site? We randomly selected sites for testing across all of our data centers.

Only 17 mins downtime for a HUGE upgrade. Good job DH!

Thank you all for the support here during this maintenance window. It is encouraging to my support staff and to our engineers as we work towards providing you with a state of the art hosting environment. We are confident in the series of network improvements we have in line.

@Ralph

ashcore.net

Is traffic being routed differently now?

Also, I lost access around 12:20pm EST. It came back for about 5min but has been gone since.

Hmm, Interesting, accessed my site through a proxy server and it came up, but still no access from here … I’m assuming IP’s wouldn’t have changed /.. perhaps it’s just Canada’s network.

lol

Could anyone shed some light on how what was done may have affected that … if it even could have??

Jon, it works for me, but it’s only a parked page.

Thanks Coyote, I haven’t gotten around to getting my actual site online yet, just use subdomains for development, etc, etc, etc…

On the phone with my ISP (waiting for a rep) to see if they can help trouble shoot this connection issue between here and there.

@Jon I am seeing the same thing as Coyote. Are you getting an error or is it an issue where your data is no longer showing up?

Good deal. You might try a traceroute to see where it’s failing.

@Jon we have optimized out network routes and we are working on getting up our redundant upstream provider. If for some reason your connection is locked in with them it could be the reason you are seeing the site as down. I don’t really think that’s possible but it’s a theory.

All my sites were affected with the fuckups at Irvine. Now, they say that they are upgrading the LAX Data Center. And my sites are yet again down. It would be helpful to know what data center you are in..

Site is down for the last 10mins…whats up with that?

oh dear

sites are down too,
Is their a way to know which data center affects your our sites ?

@Tyron your apache was down on your PS, I just started it for you and your site came up. @Balint, I need to know your site to help.

@Ralph, here’s the trace route from my location. Hopefully this can help you narrow it down.

Really appreciate all the work you folks are doing btw :)

1 2 ms 2 ms 2 ms 172.16.106.254
2 8 ms 8 ms 11 ms 10.125.97.129
3 19 ms 19 ms 15 ms 24.156.150.81
4 10 ms 10 ms 11 ms kitchener2.cable.teksavvy.com [69.165.168.226]
5 15 ms 19 ms 15 ms kitchener2.cable.teksavvy.com [69.165.168.225]
6 21 ms 21 ms 24 ms 10gigabitethernet1-3.core1.tor1.he.net [216.66.36.97]
7 34 ms 29 ms 27 ms 10gigabitethernet7-3.core1.chi1.he.net [184.105.213.150]
8 51 ms 51 ms 53 ms 10gigabitethernet3-2.core1.den1.he.net [184.105.213.86]
9 76 ms 76 ms 77 ms 10gigabitethernet1-4.core1.lax2.he.net [72.52.92.38]
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.

@Ralph, thanks, I think you just fixed it xD
Good luck with the rest of the improvements :)

I think a four hour window is much too wide. Imagine from the clients point of view Dreamhost! One hour is acceptable, but four hours! I appreciate that you are upgrading but you should break it down to smaller chunks. People would be happier with short down times windows even if it was happening for weeks rather than one big down time.

Balint…if they did that & then something unexpected happened, customers would be even more pissed. Maintenance windows are always at least double the time you think it will take. Shit happens & it happens more than you might think.

True,
As long as DreamHost does it’s best, and these down times are well warned before then it’s all good
Keep it up

Our site has been down for about 20 minutes now. What’s the deal?

Thanks

@Aaron site will be up shortly, this is unrelated to the maintenance but you helped me find a different issue with a recent upgrade we did with apache. Restart will fix the issue permanently.

Nightmare host is back…
It’s maintenance all the year. yu-uuuu!

Please get my site back online :( every minute is money lost

Hey, in case it helps anyone – our site went down during this time & didn’t come back up. It had been down for about 10 minutes when I rebooted apache (http://wiki.dreamhost.com/VPS#Rebooting_Apache) and everything came up right away.

down again!..

Can’t help you guys if you don’t put your domains, please provide them so that I can get your sites working for you.

just installing zc and site is now down.

and we are back

my site still down in uk

site still down

Ralph,

If it’s only the Apache issue pls check user balhau. All my sites down for more than 30 min now.

My sites are all down as well: http:www.iamthelab.com, http:www.faberuna.com, http://casa.iamthelab.com & http://www.shirtshoetiepant.com

@David site is loading for me. Traceroute to your site will help us determine where the issue is coming from.

@Balint sites are working for me. I did restart your apache again to make sure though it was recently started. You should be back up, all sites worked for me.

@Ralph,

they are up and running, Thank you! The restart worked! All in all excellent job, thank you guys (only 35 minutes downtime).

@Ralph,
traceroute still timing out after 10gigabitethernet1-4.core1.lax2.he.net [72.52.92.38] …

I’m calling it a night though (4am EST). Will check back when I wake and cross my fingers I can get some work done.

I’m actually taking a Network Engineering course right now, so I can only imagine exactly what’s going on there. You have my sympathies while trying to get this sorted out … Good luck :) And thanks again for the hard work. I’m looking forward to the end result!

@Jon was site recently moved? Our admins noticed that it didn’t appear to be heading towards our network as that is not our IP listed there. If your problems continue email support with Attn. Ralph and I’ll make sure to help you out.

@Ralph,

It’s also affecting connecting to panel.dreamhost.com … it’s not loading at all.

Hey Dreamhost, this continual intermittent downtime is bullshit. You guys may be upgrading. But it finally forced me to decide to move. You should send out an email with warning notice.

@Ralph,
Tried to send and email to support@dreamhost.com as suggested but get this back:

“We’re sorry, but your email was unable to be processed by our automatic support system:

We could not find the open support case you were replying to…”

… Not sure what to do now.

May 10th, 2012 at 3:04 amApple Drink Says:

All my sites down again.

Looking forward to some intermittent uptime later on though!

still??? why is this taking longer then usual.

@Ralph,
Just spoke with my ISP again.

Apparently, and coincidentally, it may be a routing issue on my ISPs end, as they’ve had a couple calls regarding routes through this path. It just happened to come up around the same time I was anticipating this DH maintenance … I checked with someone else in my city on another ISP and they could connect fine.

Some days I really friggin hate technology. lol

I’m calling my ISP back later when the ‘networking’ guys will be in to find out for sure.

My sites are down and I can’t load panel.dreamhost.com. Anyone else having this problem?

This may be an issue with my ISP (Teksavvy, in Toronto, Ontario). I can’t load my sites, panel.dreamhost.com, or even dreamhost.com.

But if I VPN to a US based server and try those sites then everything works. Very strange.

Hello Dreamhost,

It is currently 8:20am EST (5:20am PDT) and my web site is still not operational.

I am starting to have genuine concerns about the frequency of downtime that I am experiencing while utilizing Dreamhost. It seems as though my site becomes non-operational for an extended period of time almost as frequent as once a month.

Other than the high frequency and duration of downtime, I am happy with my Dreamhost experience.
Please keep up the good work.

A traceroute via TekSavvy, Toronto, dies somewhere in LA at he.net

Hello,

I’m still having issues with my site after several hours. Updates do not show and these updates are really time-critical. At least make a statement, have support actually reply me or something, some of us have people to account to.

Other than this issue surfacing for the first time ever, I have to say I love Dreamhost. But please do get this fixed ASAP!

Website down. Updates would be mighty useful DH

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours
7:00 pm

Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours

Posted (May 2nd, 2012 at 7:00 pm PST) by Ralph

Previously this was to be done on Wednesday May 2nd, but we moved it to the 15th.

On Tuesday, May 15th, we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. The fiber ring is being tested to minimize errors and to ensure maximum throughput. We are giving ourselves a two-hour window starting at 10pm PDT Tuesday May 15th. Service affected will be web and email. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup, so this should only affect a small subset of customers. This maintenance should not take the full two hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.

The DreamHost panel is up and running again. Sorry about the inconvenience!

Update May 15th 11:26pm PDT We are continuing to work with our fiber ring provider to find out regarding the unexpected issues during our previous attempts. We will update this post again when we have more information. All web hosting services are still up, and we’re looking into the downtime the DreamHost panel is experiencing. Sorry about the inconvenience.

Update May 15th 11:03pm PDT We had to roll back our second attempt as well, and are currently talking to our fiber ring provider to find out regarding the unexpected issues. We will update this post again when we have more information. All web hosting services are up, but we’re looking into the downtime the DreamHost panel is experiencing. Sorry about the inconvenience.

Update May 15th 10:57pm PDT Our second attempt for the maintenance is about to start. Please see above for the details on possibly affected services. We’ll monitor service status and will post periodic updates as the maintenance progresses.

Update May 15th 10:51pm PDT We rolled back the first attempt and looking into the unexpected downtime it caused. We’ll update this post again with further developments and maintenance schedule as soon as we can. The maintenance window is still in place until midnight tonight. Thanks for your patience!

Update May 15th 10:39pm PDT We have started the maintenance of the fiber ring between the two Los Angeles based datacenters. We will post periodic updates as it progresses. We appreciate your patience during the process and apologize for the inconvenience it’s causing.

This entry was posted on Wednesday, May 2nd, 2012 at 7:00 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

155 Responses to “Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours”

How do I find out if I will be affected? I don’t see an option / applicable category in the Contact Support categories.

Someone please Hold Me :(
I have been a dreamhost customer (mostly happy) since November 2006, a couple of days ago my sites went down, I got a an email explaining the downtime,( my server was being moved to new hardware ) and thought they would be up again within a few hours. 3 Full days after the fact all my sites are still down, the sites themselves are not that critical, a webcomic, my art and Design Webpage, and my Web Development Site.Now, if the problem was just that I couldn’t send some people to my sites, it wouldn’t be that bad, I mean 3 days + about 4-5 other days a year that they are down is not that big deal for the price, I know you get what you pay for.

This is what’s wrong: As a web developer I usually use my dreamhost account to test bits and pieces of code, I haven’t been able to do this since I can’t access anything, I am staring at my computer right now with a mix of anger and impotence since I have a client that needs something and configuring one of my computers to run as a server to emulate what I need is going to take me at least an hour. Then there’s the actual support that I am getting, no phone support ( try getting a number out of this site), and they are ignoring my ticket, if you are curious, here’s my panel info:

Current Outage: DreamHost server Microscopium being moved to new hardware (updated 44 mins 43 secs ago)
Hello!
We’re very sorry for the continued issues you’ve been having with your service. As you are already aware, the server you’re on, “microscopium” has been having stability issues. The machine had a hard drive failure and we’ve moved it to new hardware. We are in the process of copying data to the new machine now. Unfortunately, during this process the new server has been experiencing higher than normal loads under the increased strain of the data copying over, in addition to its normal functions.
We’ve upgraded the servers network connection yesterday so the restore is going a lot quicker, but that didn’t do anything to improve the stability, unfortunately. Our admin team has been trying several things through the night to get this server running smoothly again and will continue to focus on the machine until it is working like it should. I am afraid the machine has become unresponsive at this time, but our administrators are working as quickly as possible to get it up and running once more.
Sites on the server are still timing out or giving a ‘bad_httpd_conf’ error (because Apache updates are failing due to the load). We’re doing everything they can to improve the stability on this server while the restore completes. We appreciate your patience while we get the remaining issues on this server worked out. We will be updating this announcement again shortly once we have any new information on the situation. The amount of support this issue has generated has gotten us a little behind on replying. If you submitted a support request regarding this issue, we are doing our best to get back to you ASAP.
Thanks!

DreamHost Support Team + support@dreamhost.com

Really, that’s the best you can do ?

Do you maybe have another server I can use to put up 3 little webpages that explain that my sites are down and that they will be back up again when my host (thats you) figures the problem ?, how about temporary access to an account where I can do some development no?, ok how about you treat me like a person, you know I have a name ? no, how about telling me how important I am as a customer, I mean 6 Years is a long time for a relationship, No ?

What hurts is that as a developer, every now and then I get someone that asks me where should I host, and I usually send them here, so there ,If your are curious about dreamhost, this is what happens when things go wrong, they ignore you, they take their sweet time and hide, and leave you no other option than to post in a non descript forum yeah.

Keno Leon

P.S. I am going to post this on every place I can until someone from DreamHost gives me a call and apologizes/explains/helps me, in the mean time I believe I need to reconsider my hosting choice.

I’m in the same situation also, and wondering how many others are…

My sites have been down continuously since April 28, and all I’ve had to console me are the somewhat rambling updates in the Web Panel. After 90+ hours of downtime, seeing a new update this evening that plainly states “I am afraid the machine has become unresponsive at this time” isn’t reassuring. I’m not an engineer, and don’t pretend to understand the complexities involved here, but surely there are methods of bringing a site back online sooner. If a machine is having problems for four days, can’t customer’s sites be distributed to other machines? What if this server is problematic for another day, or two, or a week? How long is Dreamhost comfortable with leaving their customers offline?

are you sure you moved maintenance to monday? site down for the millionth time

the site is down right now. it seems to happen FAR MORE than any other webhost i’ve ever used. disappointed with dream host

I’m sure, going beyond idle moaning and bitching about DH always being down, people are actually migrating their businesses somewhere else… a more reliable, responsive, and customer oriented host. I think I’m teetering on the edge of doing just that, but am not sure who out there is the suggested hosting company. Can someone share?

the site is down right now. it seems to happen FAR MORE than any other webhost i’ve ever used. disappointed with dream host

This is valid for my site too.

Please take urgent solutions to these problems!!!

Dreamhost officers do you hear us ?

This is why I cancelled VPS service, if this goes in that way, I will move my site to another company. That costed my company very big amounts. Will you pay me for that problems ???

May 3rd, 2012 at 7:37 amphiya@dreamhost Says:

Yusuf, the machine you’re on has been up for 110 days, although the apache instance you’re on was recently restarted 4 hours ago. Looking at apache status, I can see that most of the connections to http are to your domain. There’s a few IP addresses that are hitting your site excessively. I’ve sent you an email directly with details.

Keno, sorry about your experience with our service as of late. All I can say aside from sorry, is that we’re committed to offering a more reliable service, and are in the process of making major changes to our infrastructure to achieve just that. No one likes instability, including us!

PS: I will directly pass your concerns to our support team. I can probably get your quicker support if you included your domain in the website field.

Phiya
Data Center Operations Admin

Our sites are down. Again. Irvine? Everyone please submit tickets so that they address this ASAP.

same here, sites down… on ps83992
Could you please advise me asap on the issue.

Thanks,
Teo

this place is a joke, losing clients everyday…

back up :)
Thanks DreamHost !

My sites have been down for 3 days. I am losing money and no response. Just that they are working on it…

That’s not enough. We pay for that service and they are not reliable.

This is so disappointing. I lost my trust.

This is not the first time my website is down for a long time. Since I moved my website to dreamhost they stopped working a couple of times already. This doesn’t make me feel very comfortable to say the least.

May 3rd, 2012 at 1:38 pmAnastasia L Says:

Yup, microscopium still down, day 6. Is it the only server having this problem, or are some of you on other machines?

Last I heard from support they expected it might be another 5 days or so to fix. I also don’t understand why they haven’t just been able to move us to a server that IS stable, or at least put us all into some sort of maintenance mode with a prettier error message.

@Anastasia L : Other MySQL servers are also causing troubles. I lost my database tables and can not restore a backup.

….”this should only affect a small subset of customers”….

Yeah right, to bad I am often part of the small subset.

Currently looking for a new hosting party. If I find a good alternative before the problems with dreamhost are fixed; I am out of here.

Been down three days going on four. Tech support has been a no show and no one is communicating. Perhaps it is time to leave dreamhost. I surprised there hasn’t been some sort of class action suit filed against them.

We have been down more than 3 days on KuwaitCity as well. I have a half dozen or so customers out. 3+ days. No callbacks. No information. Embarrassingly bad performance Dreamhost.

Having the same issue, they must be sleeping on the job. I’m up for class action. Could be fun

May 3rd, 2012 at 3:08 pmdreamhostishell Says:

They must have wax in their ears or else they are all blind. I’m pissed and I am trying to refrain from cussing. As soon as I can ftp, I’m getting my files and leaving dreamhost. Did you know they got an “F” on the Better Business Bureau website?

Current Outage: DreamHost server Kuwaitcity being moved to new hardware (updated 1 day 1 hour ago)

Kuwaitcity has been down for 3 days and there have been no status updates for over 24 hours. Server migrations don’t take that long. What is the delay?

Same for me and no reply to my emails. This is truly disappointing.

I’m on astana and one of my domains hosted on it is down.

I’ve complained at how slow their mysql servers are since i’ve been with them. The last several months service has been horrible and sites have been going down for days at a time. All we ever get is a we’re working on it.

I’m sick of this……Ya know Dreamhost any real hosting company would have RAID setup so that if one server went down a backup server would automatically kick in so there was no chance of downtime for a site.

What I do find interesting is that this domain has never gone down. At least not that I’ve ever seen.

Keep up the great work guys… even with all the complainers above…

My sites are down, can not connect to MySQL.

When will they solve the problem?!

May 4th, 2012 at 7:41 amdreamhostishell Says:

up and down up and down, damn you guys are clowns

no faith in DH, they will botch this upgrade and we will all go down yet again…

May 4th, 2012 at 8:41 amdreamhostishell Says:

That’s it, I’m sending a nice little press release out to my extensive media list. They had their chance, they blew it.

May 4th, 2012 at 8:43 amdreamhostishell Says:

Going to a new hosting company. I will never use dreamhost again and I will tell my customers to never use dreamhost. There are plenty of affordable places that actually have 24/7 support and a number you can call. So long dreamhost, thanks for nothing.

May 4th, 2012 at 8:51 amdreamhostishell Says:

These people are fricken idiots. It was up momentarily and now its down yet again. F You dreamhost.

May 4th, 2012 at 9:41 amdreamhostishell Says:

I just signed up for a new provider, I ordered and 2 minutes later someone called me. So far so good, things are looking good.

im not sure why there is always some sort of outage, my sites down again, i m fairly sure my sites been down at least 5 or 6 times in the last few months, im not sure how this is acceptable, and damages my business everytime it happens, yet all i can do is email dreamhost because talking to them would apparently cost me additional money…go figure, so do i have the right to charge them money everytime my sites down.

May 4th, 2012 at 10:30 amdreamhostishell Says:

You could yell at the top of your lungs and they still wouldn’t hear you

Site down

My site went down sometime today and is still down this evening. What’s up?

SITE DOWN FOR 8 HOURS!!!!!. HELP ME PLEASE.!!!!

I have about 40 websites hosted with dreamhost, they’ve been down since wednesday 2nd of may. I have 40 angry customers who are losing money and have been calling me every 10 minutes for the last 3 days.
Dreamhost support took ages to get back to my mail and still no solution and no ETA. This is unaccetable!
Can someone please suggest a more reliable hosting company? Time to move away from this unprofessional service.

May 5th, 2012 at 5:50 amdreamhostishell Says:

Been down almost a week. They don’t get it and they don’t seem to care. I urge everyone to leave, Dreamhost is a sinking ship

May 5th, 2012 at 8:23 amDan Poynor Says:

Ugh. Sites have been down for nearly six hours.

Sometimes Dreamhost does hardware replacement which they don’t post about on dreamhoststatus.com apparently.

I’ll definitely leave dreamhost if they don’t fix the problem before monday.

May 6th, 2012 at 2:58 pmAmerican Ex pat in NZ Says:

My husband’s websites in New Zealand (about 30, on DreamHost server Kuwaitcity ) are down and have been for close to 6 days. I’ve called the Brea phone from here (NZ) on his behalf and have left multiple messages. He also is getting concerned / angry calls & emails from clients and has lost the business of a new client. It’s impacting our customers as THEY are also losing revenue in these sites being down. The lack of an ‘uncanned’ response by Dreamhost is very unprofessional and disappointing.

All my sites were down over the weekend.
This morning (Sunday night, USA time), they were working again.
Then two hours ago, the server is COMPLETELY empty…..all files gone….nothing left…

May 7th, 2012 at 2:02 amdreamhostishell Says:

looks like they deleted everything. morons

OKAY, it seems to be restored. All systems are go. Phew!

Site down for 5+ days on server kuwaitcity. Bizarre that there’s no acknowledgement of this outage on this site dedicated to Dreamhost server status; it doesn’t give me confidence. Messages from support have been confusing and unhelpful.

I know that this is not an official way to contact you… but might I suggest doing scheduled maintenance at a time that is slightly later than 10 pm PST? I mean I feel that even 11pm and later is more beneficial, and seeing as these users (correct me if I am wrong) are in the United States, these hours are low-traffic hours for clients. I definitely understand how updates are to be done at “downtime”, but I feel that even 10pm PST is too early in the night. Just my thoughts. I hope that the fiber switchovers will help maintain structure stability as well through the data centers. I know it’s a pain to keep systems up while actually replacing hardware.

Thanks for your posts,

Alex

@Alex Silver,
DH has customers all over the globe, so no matter what time they do it, some will be affected later in the night (for me it’ll be 1am EST.) and others will be affected during the afternoon/evening hours.

For whatever reason they have chosen 10PM, perhaps it’s more right in the middle time wise globally. Considering what they have planned for the end result, I’m certainly not going to complain (not that you are) about a margin of downtime for the greater good overall.

Maybe I’ve just been lucky, but fortunately the server I’m on has never seen downtime due to hardware (knock on wood). I feel for the people who have had a lot of problems, but that’s the nature of technology and hardware, nothing is 100%. Being a network administrator is not an easy job, especially with the supply and demand of today. Anything can happen at any point, but I think the network team at DH does the best they can with the resources/technology/hardware they have available to them at any given time, to keep things running.

I’m really happy to hear and see these core hardware upgrades taking place, and look forward to what’s ahead :).

Just something to keep in mind ;)

Keep up the good work DH!

@Jon

Agreed. I understand the nature of the maintenance, and I was being cautious not the presume that all of the Dreamhost customers are in the U.S.

I’m glad to see that they are being *proactive* and not *reactive* to volume needs and forward preparation. Seems like they see a potential for more volume of users and domains being hosted. I do appreciated it. I work in Education Technology at the network level, and I know that downtime kills anyone at any time (even though we question why staff is entering grades at 3 am). I chose to use Dreamhost for personal websites because of their local business nature (I live very close to one of their farms) and their up-front products. I hope that they get these upgrades on the road so we can enjoy their great service.

Alex

Hi from Spain,

I would like to know if there are problem because my site is down.

Regards

Jorge

May 11th, 2012 at 3:06 pmAmerican Ex pat in NZ Says:

kuwaitcity server has been down for 10 full days, going onto the 11th … I have called and left messages on the toll free line (which isn’t free when calling outside the United States) requesting any nugget of current and realistic information on this outage that we can get and in turn, pass on to our customers.
The silence from DreamHost thus far has been deafening.
I realize we (as a business) are small in comparison to some in the global network that DreamHost serves, but we are now truly feeling like the forgotten minority ~
Our fears of having to start over or shut entirely are very real at this point as existing customers are wishing to close their accounts and potential new business has gone elsewhere …

I can’t seem to find and update on anything that matters. Now it appears many are having all sorts of issues and nothing on the blog about it. Keeps going up and down. Pingdom is a great service. Love trying to work and cannot.

Anyone elses sites down? :(

Tuna fish is good sometimes. Tuna boats!!!!

You know what would be really bad? If Dreamhost just didn’t communicate with us about their latest rash of outages. Oh, wait.

Does anyone know if this affects dedicated servers too. One of my sites it loading extremely slowly. The speed was restored for a short time and now it’s loading excruciatingly slowly.

Thanks dreamhost, just got a call saying 3 months of content on the 40 sites I host has been lost. Why do I even pay 200 a month for a vps?

May 13th, 2012 at 11:50 amThe Exorcism of Dreamhost Rose Says:

Dreamhost slowness, CloudFlare down: PIG GOD!

May 14th, 2012 at 10:59 amJonathan Tonge Says:

Down. I have something I’d like to introduce. The Dreamhost “WE’RE UP” messaging system. I’d suggest a “WE’RE DOWN” system but I don’t have that much time to go through all those emails.

All my domains are currently down. Emailed tech support and received a reply that they (my domains) were expired. I know this not to be the case. From my end, I cannot load the panel page, or even Dreamhost’s home page.

May 14th, 2012 at 1:57 pmJesse Miles Miller Says:

mysql server synthesizer cannot be reached.

synthesizer:yarrell I ma getting a Mysql Database connection not available. Is this the same or different? as the May 15 issue?

My site just went down as I was uploading new content. Please advise when I may expect to see the site back up and running again.

All my sites are down (including the panel), errors a gogo. Is anyone there? Could you please update us? Knock?

my mail is down too. I’m on Texans. Is there a human at the end of this line?

Much to my frustration, we’re past the 24 hour mark on server downtime here. Diablo III was released last night and I have community users who are desperate for access to the site. Get your shit together, DreadfulHost!

My email is down–all accounts.

Same here. My mail just went down.

Here too, all email accounts just died

Actually strike that, the emails back up again, musta been minor blip.

I host 13 sites an they have all been down for more than 2 hours.

The moment I find some time to complete my transition to another VPS hoster I’m getting the hell out of DreamHost. Experience terrible performance for my Wiki for the last week.

All of my sites are down, getting these errors:

“Coppermine critical error:
Unable to connect to database !

MySQL said: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110″

and

“Error establishing a database connection”. Any updates? Very frustrated.

I love being in that “small subset of customers” that seems to be impacted by every single issue Dreamhost has.

I’m getting some ridiculous xml parsing error. I run my business from this site. This is completely unacceptable.

Our site has been down since 8:52am. We use a reporting tool that relays uptime and downtime via text message. I have been getting text messages all day. Up, then down, then up then down…but the whole time it has been down. Sweet!

jared – I don’t think your problem has anything to do with Dreamhost. It says the error is related to your Google +1 button.

Aaron – if you’re right, that would be the biggest coincidence ever! I haven’t updated or logged into the admin section of my site for a month, and the site and g+ buttons have been working the whole time! Now, even if I remove them, the site doesn’t function properly!

Actually all my emails died again, for a couple minutes, now back up again. Seems very up and down today, might be worth reporting.

i keep having to reboot my vps just to get it back up and running. not happy once again for like the 50th time this year DH!

One of my sites is down that is resolving to this IP: 208.113.237.7

why don’t you just put my DB in the same data centre as my server so I don’t have to deal with this every week?

@bob – That is something we have been gradually doing. If you can let me know your domain, I can double check if your services are already in the same datacenter.

Tommy wrote:

“What I do find interesting is that this domain [dreamhoststatus.com] has never gone down. At least not that I’ve ever seen.”

dig dreamhoststatus.com
96.126.109.82
dig -x 96.126.109.82
li366-82.members.linode.com.

Of course, it makes perfect sense for Dreamhost to host this domain on someone else’s network and hardware.

@andrea i’ll request through the panel. no need to put those details in a public forum. thanks.

c/p is down

panel as https://panel.dreamhost.com/ not working.

Error 101 (net::ERR_CONNECTION_RESET):

All sites are *not* up – mine are down (dedicated server) and no access to panel to report a problem. Have tweeted @dreamhostcare, fingers crossed.

BTW, I don’t have a front-end/DB split as described as a potential issue in this post.

My dedicated server is down as well. Database is hosted on the same computer. WTF? It’s laughable how many times I have experienced in the last 6 weeks. Absolutely unbelievable. My aunt’s 10 year old Windows XP computer has a better uptime record than my dedicated server hosted here !

can’t even report the damn issue because the deamhost panel is down too … freak i am pissed off!

DH panel not working and sites unavailable. Dreamhost is simply not what it used to be – I have had nothing but issues with DH over the last 6 monts. Your hosting service quite frankly is in a terrible state!!!

Customer since 2004. Who cares…

I use Linode as my backup server. I never have outages there. It’s like Linode.com is a dream host, or something. It just sucks that I have put my faith in Dreamhost to the point where I am deeply invested and I really don’t have the expertise to run my own server with all the bells and whistles or I would move.

Down here… Please help…

Going down during scheduled maintenance is aggravating enough, but having my site down when this status site says it should be up is beyond frustrating. With all the downtime lately (and a fair amount of it due to DH issues and NOT DDoS attacks), I would REALLY know how to get refunded for the downtime. DreamHost, your own site says, “Our 100% uptime guarantee means we take uptime seriously.” Prove it.

My dedicated server is also down and it’s coming up to 8am here in London. Maybe we should find out who hosts this site as it never seems to suffer any down time?

Yeah, so “All web hosting services are still up” is simply not true. In my case, it’s about half true.

My shared server is down. My VPS however is running, good job!

I am down as well. I can’t even access the control panel or tech support for Dream Host.

Andrea/Other DHers: An update through @dhstatus and/or this status page would be really appreciated. I can see lots of tweets going to @dreamhostcare but that account hasn’t tweeted in the last 4 hours (is it manned 24 hour?)… Thanks!

Almost midnight, the maintenance window end, and still down…

I agree with PatrickD. When do we get a refund for all of this downtime? It’s a major headache I deal with (seemingly at least one a week.)

“so this should only affect a small subset of customers” – all our 6 clients websites

Jim: I think DH use Linode for hosting this page.

… by the way, all email seems to be down too. Even so, I don’t see the point in freaking out on this forum. It will not do any good and only serves to aggravate the people who are probably working very hard to get it back up and running. I believe DH are trying to sort out their issues but change has to come from the very top of the company. This is where the problems seem to be: in the areas of planning and strategy.

Our VPS is down. Our web control panel is unreachable. What to do?!?!??!

website and everything related is down. even the DH panel not opening?

No, sites are NOT up. Am down, and have been for awhile …

… and I can’t even log a ticket ‘cos the panel is still out!

Correct, CPP. My sites are down and panel is inaccessible as well. Please post an accurate update about the current situation Dreamhost.

this is not happening again…….. i dont think dh realizes how many phone calls i already got and will get because of this………

Yep, all of my sites are down as well.

Grrrrrrrr.

VPS sites still down and so is the Panel.

Oh ffs. It’s down again….

All of my sites are still down, and no panel either. Sort this out!!!

panel just came up but website is completely down…

Panel seems to be back up although the sites are down

Just passed 30 mins of downtime and still no acknowledgment of the outage either here or via @dhstatus… A little update saying “we’re on it” would make us feel a lot better :)

Dreamhost, any idea when this will be resolved? My site’s down and it’s just hit 9am in Europe…

Also, could you please answer Jim’s question about where this status site is hosted?

May 16th, 2012 at 12:16 amShawn Gossman Says:

ds2032 is down. I am assuming this is the cause. Dreamhost, I give you exclusive permission to email about scheduled downtime, please pretty please with a cherry on top… That would make life easier…

Al my main websites are down. I must perform a demo and send a campaign pointing to my websites ASAP. My business is being compromised!

Hi,

Website has been down for more than 30 mins now, please advice on when it will be back up? 8:16am UK time and VERY inconvenient for business.

The DreamHost panel was confirmed to be down, and it’s now again loading. Please see status updates for information.

@ Chris, Ki, Charley – what is your domain? I’d be happy to check. As far as our test domains show, the web services have been up since the last rollback around 11pm.

@Mads, Rafael Antonello, PatrickD, Jim, Jeremy – you’re not affected by the maintenance, you’re in a different datacenter. I have alerted Tech Support, and someone will get back to you shortly.

Andrea, can you grab my domain details from the email I supplied when posting my comments? My twitter handle is @crobinuk

@michael linode.com

Andrea: Also, I have the Panel set to lock down by IP Address, and it’s now telling me my IP is not authorized (it was). I went through Auth process again and it’s still not working.

@Andrea you can check by my email address I inputted in this comment for for the website url …

Andrea can you check why is my server not reachable/down ds2264.

I like how they say no sites are affected, when many clearly are. DH is not very good at communication, which is odd, because that’s their main business.

and you know better than “As far as our test domains show, the web services have been up since the last rollback around 11pm.” is true

I think I’m seeing an outage here that coincided with a loss of panel access as well. Unsure if we are affected by this maintenance.

Rather.. our site is offline, coinciding with PANEL outage, site is still offline.

Mine is down as well Andrea

why is the website still down when you have cancelled/postponed the maintenance? did you forget to plug back the old hardware?

Can you provide an ETA on when this might be back up again. AUS EST 5:20PM over here and this isn’t very good for my clients.

All of mine are still offline. They are on the Rotanev server, if that helps.

2fat2furious.com is one of ‘em.

“Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.”

Does this mean that sites will be back online in the next few moments. or does it mean we have to wait til your fiber optic ring provider works out the problem?

Andrea: I logged a ticket @ 11:47pm – #5068256. Site is still down.

Our site is still down as well, is this in relation to this? Or some other problem that just so happens to coincide?

All my sites has been down! Please help me!
My main site is http://dab.tw/ in Web Server “rotanev”.

Thanks.

The control panel is my sites are down. Can you please give a clear update on the site outage?

The control panel is useless if my sites are down. Can you please give a clear update on the site outage?

Our site is also down for about a hour now. Any progress with dealing with this fiber ring maintenance?

Panel is up, but our site is down. As comments here state – you have lots of problems, DH. Please do something.

OK, the Panel is up, but MY WEBSITES ARE NOT! (hosted on cedar).

Please, resolve this asap. All our businesses are in risk.

already moved my files to a new server. By the time dh figures out whats going on, my dns should have already resolved to my new server.

M what server are you moving to? any suggestions?

Everyone that has a site down – if you have a dedicated server this is probably due to “networking issue” at the Irvine Data Center.

Check this (new) thread for updates: http://www.dreamhoststatus.com/2012/05/16/networking-issue-at-the-irvine-datacenter/

back now :)

my website just came back up

I am back up and running, thanks Andrea

Control panel just came back. The site is still down. It’s been a while now !
Please update quickly !

May 16th, 2012 at 12:43 amShawn Gossman Says:

ds2032 appears to be up. I only paid for one hour of downtime this time around :p Good to know my $120.00 a month is going to something…erm…or is that nothing?

Site is up again.

Looks like all the downtime reports may have been due to this instead – http://www.dreamhoststatus.com/2012/05/16/networking-issue-at-the-irvine-datacenter/

Yep, up again. Thanks.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Networking Issue at Irvine DataCenter (RESOLVED)
12:53 pm

Networking Issue at Irvine DataCenter (RESOLVED)

Posted (April 28th, 2012 at 12:53 pm PST) by danwy

We have noticed connection issues in our Irvine datacenter. This is resulting in site and database downtime. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update April 28th 1:20pm PDT We apologize for the continued down time but our network engineers have reported progress and we believe we’ll be fully operational again shortly.

Update April 28th 1:42pm PDT We would like to report that all of the sites we have tested that were experiencing issues are back up and loading for us. We’re continuing to monitor things closely but it appears that this matter has been resolved. We will provide more details when we have been able to assimilate all of the feedback from our admin and networking team members.

Update May 2nd 2:20pm PDT While this issue has been fixed since our last update, we kept this post unresolved until we were able to get more information about the specifics of it. This issue was caused by another DoS attack. During the process of blocking the external origin of the attack, the core routers became unreachable and therefore blocked traffic to Irvine. We had to reboot the routers to solve the problem and return to a stable state. We have already planned on replacing these routers with much more powerful ones in the next couple of weeks to help avoid future issues like this.

This entry was posted on Saturday, April 28th, 2012 at 12:53 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

164 Responses to “Networking Issue at Irvine DataCenter (RESOLVED)”

April 28th, 2012 at 12:54 pmAngry Designer Says:

Dammit. Please update regularly.(First Post)

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?

MY sites are all down, and have been since 12:25. First posting?

Guys, this is the 6th time in a row that my site is down. I am already getting killed by my clients….

If I wanted a host that was spotty and went down all the time, I’d host on a server in my basement. Come on, Dreamhost!

Thank you for choosing Dreamhost where we guarantee 20% up-time

April 28th, 2012 at 12:56 pmNoxiousGames.com Says:

Noticed this around 12:20 also. I thought it was my server specifically down but I guess not. (Found out trying for subversion, ftp, and web)

This is getting to be too much. How many times has it been this month? You need to give out a credit for how often the hosting has gone down.

But that 20% guarantee is really just an estimate. YMMV.

Site’s down for what, the sixth time this month? Once DH kicks back on I’ll be backing up my domain and switching to Bluehost. While DH is cheaper, outages look incredibly unprofessional to clients.

BURN IN HELL DREAMHOST…. IM TOTALLY SICK OF YOUR TERRIBLE SERVICE…

Why does this keep happening??????????

Every time I give Dreamhost another try, they fail. And I keep saying, “Oh that will be the last time.”

by bye dreamhost

waiting patiently…

And I increased my resources
–‘

Maybe is time to say goodbye

It is only the Irvine DataCenter, I never hear downtime at their other data center. How on earth can I get moved there!

http://dreamhost.com/partners/alchemy-communications/

http://blog.dreamhost.com/2010/04/01/datacenter-innovation-from-dreamhost/

First update 30+ minutes after service issue occurred???

This is absolutely terrible service you are supplying us lately. All that seems to happen lately is downtime, downtime, downtime come on sort your systems out please

Take this Irvine DataCenter and put on a trash!!! WTF!

Unfortunately we started to negotiate with Rackspace to migrate our servers.

Let’s all transfer to HostGator! Here’s a 25% off coupon: save25withpat. All proceeds go to our charity in Haiti (www.rfahaiti.org)

Here we go again, same old problem.

April 28th, 2012 at 1:01 pmbuythiscomputer Says:

***********************************************************************
***********************************************************************
***********************************************************************

CAN WE BEAT THE RECORD ? MORE THAN 1000 COMPLAINING COMMENTS THIS TIME ?

***********************************************************************
***********************************************************************
***********************************************************************

WHEN WILL DREAMHOST GIVE A REAL EXPLANATION ? NOT THE USUAL BULLSHIT…

***********************************************************************
***********************************************************************
***********************************************************************

You don’t know how much i loose with this, where are your genious ingeneers……………….

It’s a joke?
It’s a joke?
It’s a joke?
It’s a joke?
It’s a joke?
It’s a joke?
It’s a joke?
It’s a joke?

Can we get an estimate on when this might be cleared up? I’d cleared out my saturday to work on a project and this has pulled the rug out from under me.

Maybe they should bulldoze the Irvine data center and build a new one somewhere else.
Seems like it’s located on an ancient Indian burial ground and the spirits are angry. Either that or the string broke between the two tin cups.

Can some one please nuke Irvine and take it out of its misery?

Thanks.

SWITCHING TO A VPS IN IRVINE HAD BEEN THE BEST CHOICE OF MY LIFE.

WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.
WHAT HAPPENS IN IRVINE, STAYS AT IRVINE.

Dreamhost prices are very low and people are friendly, but the frequent problems demonstrate that there is no free lunch. Does anyone have any suggestions for a reliable hosting company?

It just doesn’t end. More outages than an Episode of Project Runway. Please somebody recommend to me a better Hosting company.

Gosh, Dreamhost has more downtime than uptime.

Well…just make sure you read the fine print in their 100% uptime deal…

http://cgi.dreamhost.com/hosting-100-percent-uptime-guarantee.html

Guys, looks like I am able to ping the domains right now! Check it out

I just moved to a VPS for my clients and we are getting ready to launch on Monday with new features and now I read all of this. WOW DH you get one chance with me, and this ones it.

So tired and angry about all of this :(

I will chargeback the shit out of DreamHost and advise you all to do the same!

April 28th, 2012 at 1:07 pmAngry Designer Says:

Mine are coming back up

This is just stupid. Are you guys all rookies?

I’d just like to say this is the first time I’ve experienced Dreamhost downtime at a noticeable time (by myself in development, fortunately, not a client on launch).

Looks like my servers are up again, in under an hour. Given the overall service Dreamhost offers, I find nothing to complain about.

April 28th, 2012 at 1:08 pmAngry Designer Says:

I’m sure everyone here uses this, but UptimeRobot.com is free and amazing.

Why cant they fix things so that this does not happen every day…..website down/server down!!!

I can only work on my site on weekends!!(day job during the week) This is just terrible!!

Have you guys consider moving everyone from Irvine data center until you figure out what the heck is going on? A lot of people depend on their websites for their livelihood. How about we start taking that seriously?

When will they lean tin cans and string for your network is so 1980.

Serves you right you DANG, DIRTY TROLLS.

Hey, look! It’s back! Someone tied the strings together! My sites are back, and just as sluggish and slow as ever!

But for how long?

I sure hope everyone who has problems in the future floods the support team with trouble tickets every time there’s a problem with their site – maybe the shear mass of complaints will motivate them to upgrade to equipment that’s at least based on 2008 technology or newer.

I haven’t had to pay dreamhost in three straight months… Remember, per their policy, send an email to their support through panel.dreamhost.com (if you can get there) requesting a months credit.

I really regret saying this. I was referred to dreamhost by two of the most intelligent and sophisticated technologists that I know. At the time, 16 months ago, I think it was a good recommendation. Unfortunately for me, I have the worst timing in the world (got heavy into stocks and real estate in 2006 ;-). I am now investigating my options. I am happy to pay for service. I will pay well for quality service. Even though Dreamhost is currently free for me, it just ain’t worth it.

I am looking for something seemingly simple (I have done the sysadmin gig for LAMP stack, so I know it ain’t always simple and easy)… I want a provider who I pay such that when someone goes to a URL that accesses a DB, something I define displays. That is all I want.

Here endeth my rant.

Seems like the connection is back up. Thanks for the quick response DH.

i thought cloudflare was supposed to fix all this? it does nothing!

April 28th, 2012 at 1:14 pmEnough is Enough Says:

……………………………………..________
………………………………,.-’”……………….“~.,
………………………..,.-”……………………………..”-.,
…………………….,/………………………………………..”:,
…………………,?………………………………………………\,
………………./…………………………………………………..,}
……………../………………………………………………,:`^`..}
……………/……………………………………………,:”………/
…………..?…..__…………………………………..:`………../
…………./__.(…..”~-,_…………………………,:`………./
………../(_….”~,_……..”~,_………………..,:`…….._/
……….{.._$;_……”=,_…….”-,_…….,.-~-,},.~”;/….}
………..((…..*~_…….”=-._……”;,,./`…./”…………../
…,,,___.\`~,……”~.,………………..`…..}…………../
…………(….`=-,,…….`……………………(……;_,,-”
…………/.`~,……`-………………………….\……/\
………….\`~.*-,……………………………….|,./…..\,__
,,_……….}.>-._\……………………………..|…………..`=~-,
…..`=~-,_\_……`\,……………………………\
……………….`=~-,,.\,………………………….\
…………………………..`:,,………………………`\…………..__
……………………………….`=-,……………….,%`>–==“
…………………………………._\……….._,-%…….`\
……………………………..,<`.._|_,-&“…………….`

Thanks for restoring service.

This is the first time I’ve caught a major outage not related to Cloudflare — although I DID get a CFlare err msg on one of my sites that no longer uses CFlare (good riddance).

And, since my previous hosting company started out great but fell apart after about 6 months, I regularly tested a couple of DH hosted sites using Integrio Uptime Scout for months before switching to DH (with several months remaining on the old host, good riddance there, too).

Anyhow, DH was really reliable in my previous testing and, so far, seemed to have been quite reliable.

That said, like everyone else, I can’t have my business site or client sites on a host that isn’t truly available 99.9% (and I’d really prefer more — after all 99.9% still allows around 8 hours of downtime a year — and that’s enough to lose a fair number of clients.

Buck up, DH, and please try to make sure this doesn’t happen again.

Wow down a whole ten minutes… Think anyone noticed…

Thanks, back up. I have a question though… why isn’t status.dreamhost.com synchronous with dreamhoststatus.com? It seems if you have 2 sites dedicated to the same thing… ah, nevermind. Just please keep our sites up, thanks!

Mines still down, bahhhh I need my website

|^^^^^^^^^^^\||____
| The STFU Truck |||””‘|””\__,_
| _____________ l||__|__|__|)
…|(@)@)”””””””**|(@)(@)**|(@)

DOWN AGAIN :>

Well, I spoke too soon. It’s back down.

NOT HAPPY.

I have been on Host Gator and switched to DH to avoid this exact thing. I own the company to which the site is down, its been an hour and we have lost a lot of money.
I dont understand why this kinda thing keeps happening, its like a bunch of pot heads barely holding onto their shit…
On top of all of that we have a VPS… Our techs were in the middle of upgrades and about to push update and then BAM all work gone.
Whats the solution here, I NEED TO KNOW!. I cant run a business like this. Anyone know of a host that doesn’t crash weekly of daily…?

April 28th, 2012 at 1:19 pmAngry Designer Says:

back down :(

DOWN AGAIN. F*CK

Just when you thought it was safe to go back and update your site…DH outage strikes again…Error establishing a database connection

Didn’t tie a strong enough knot in the string I guess… try one of these http://meritbadge.org/wiki/images/5/5d/Six-Boy-Scout-Knots.pdf and maybe I can get some work done today.

@jwwicks — Is that double reverse sarcasm?

Because, clearly, it’s been down way longer than ten minutes and clearly lots of people have noticed. I’m expecting client calls to begin any time now…

┏━━━┓╋╋┏━━━┓╋╋┏━━━┓
┃┏━┓┃╋╋┃┏━┓┃╋╋┃┏━┓┃
┃┗━━┓╋╋┃┃╋┃┃╋╋┃┗━━┓
┗━━┓┃╋╋┃┃╋┃┃╋╋┗━━┓┃
┃┗━┛┃┏┓┃┗━┛┃┏┓┃┗━┛┃
┗━━━┛┗┛┗━━━┛┗┛┗━━━┛

Back up for a few minutes, and then down again!

My site is working but piwik is showing me installation window wtf:D

And yes, it happened again with dreamhost:) Start thinking about giving us some ‘happy gifts’!

@Kevin, on the downtime policy, exactly while you all were here posting you should have been putting in a support ticket or you don’t get credit :)

Then come here an post…

Rats, spoke to soon, down again… I’m glad it’s Saturday or I’d be getting an earful… Least my tickets in :)

Jw

Yep, damnit back down. As for other hosts out there… as long as you are paying $10/mo or less for “unlimited” space/domains/etc., there’s not too many options out there. If any of your sites are truly making money, then perhaps it would be wise to look into some managed hosting.

EVERYTIME THE POST SEEMS LIKE RESOLVED, DOWNTIME AGAIN.

FIX: DON’T SAY IT’S RESOLVED.

Still down, and Dreamhost has ignored my requests for reimbursement since the start of the year putting in tickets when I first noticed issues every time. Downtime is okay. Not backing your guarantees as promised is not okay. The first year was great, but I’ll be moving on now.

I am quite amused that people keep asking for refund. haha. Losing your credibility as a business when your site is down that often can’t be undone with a simple refund. Sigh…. I was hoping to postpone the move as long as possible. Seems it is inevitable at this point :(

April 28th, 2012 at 1:24 pmAngry Designer Says:

Im currently signing up for Bluehost, anyone have any experience with that?

@Angry Designer … I have heard good and terrible things about BlueHost. The only hosts I can recommend are WiredTree and VPS.net

Damn, this really sucks, downtime even in the cloud computing era….

@TK sort of, it was down then up then down so I’m hoping anyone trying to get into the VPS’s or Dedicated Server’s is getting through in the up swing part.

Jw

On the two week free trial. Looks like a frequent problem. While GoDaddy gets bashed for all kinds of things, they sure don’t have downtime like this. Hosted with them since 2007….. Never EVER experienced downtime.

Will definitely be sharing this info with my client, especially since because of this downtime my work is late. Sorry dreamhost.

I am SO sorry I picked Dreamhost! This hosting is terrible.

@amrit, it doesn’t have nothing to do with cloud computing but with routers and shit.

Stop spending money on parties, tours, free food for your employees and other forms of “fignity,” until you go for 3 months without downtime.

3rd time this month.

Nice.

I want a refund.

Seriously. You all need to compensate all of us who’s sites are down.

JL

I thought CloudFlare was supposed to prevent website outages for me. Why is my site getting a “404 Not Found” error?

Check these guys out playing Mortal Kombat 3 instead of administering the servers: https://dreamhostcareers.silkroad.com/dreamhostext/Home/Fignity.html

I never seen the Irvine data center, but in my mind, I imagine it’s an old abandoned warehouse with busted windows and a leaky roof, located on the bad side of the railroad tracks and home to a large homeless population. I think one of the homeless people tripped over the 10Mb coax cables, pulling the card out of it’s 8-bit ISA slot and caused the outage, or the birds making nests inside the Compaq 486 servers chew away at the 50 pin SCSI cables on the full size Seagate ST43400N hard drives.

The site is Down since : Apr 28, 2012 – 08:00:15
Almost 6 hours !!

@Ryan and all the rest. Rackspace, vpsnet are the only solutions that I can confidently recommend.

This really sucks …. What’s worse is we are at the mercy of Techies….. I’m missing out on a lot of business right now ALOT …. Git R done

…………………./´¯/)
………………..,/¯../
………………./…./
…………./´¯/’…’/´¯¯`·¸
………./’/…/…./……./¨¯\
……..(‘(…´…´…. ¯~/’…’)
………\……………..’…../
……….”…\………. _.·´
…………\…………..(
…………..\………….\…

YOU

DREAM FUCKING HOST

Try the 5th or 6th time this month. I’m not paying another $100/mo for a dedicated server when I have to keep apologizing to our advertisers and audience.

@T I’d agree but most of our sites are not on shared hosting, mostly VPS and Dedicated Servers and with the way MySQL is migrated to the greatest common denominator, ie the first VPS of the first DS when it goes down they all do…

RackSpace is the only business critical host I’d trust but it’s ~twice as expensive.

Jw

April 28th, 2012 at 1:33 pmJ-P De Clerck Says:

Thought a VPS solution would help. Thought there were actually more data centers. Thought I wouldn’t waste precious time and money again. Thought they became better. Thought I would enjoy my weekend.

I’m getting this error:

Warning: mysql_connect() [function.mysql-connect]: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110 in /system/database/mysql.php on line 6

You have no clue how many issues I’ve had with Dreamhost.. or maybe you do but seeing as I have so much invested and switching 6 sites over to a new host would be a HUGE pain in the ass I’ve stuck it out and stayed but as a business… I can NOT afford to have my websites offline for hours or days at a time. I’m already on their VPS service because their shared hosting is HORRIBLE but now here we are again even on a virtual server I’m having the same kind of issues.
What really bugs me.. is the lack of communication. Last update is on the 25th and it is now the 28th!!! Come on WTF???
Being someone who has worked for a web-hosting company in technical support this is just unacceptable.

NOTICE DREAMHOSTS COMPANY WEBSITE IS NEVER DOWN.. THEY’RE PROBABLY HOSTED WITH BLUEHOST! lol!

Looks like this is starting to get to be a big problem.

In the last hour I’ve had two of my clients call me to ask why their site is offline. Two others last week have already been asking me why I recommended Dreamhost to them.

Maybe it’s time to move on people, if anyone has any ideas/recommendations about better hosts post them for us.

Sort this out Dreamhost, the brand experience is rapidly becoming a bad dream. “Nightmarehost” anyone?

Recommended DH to a client last month and seriously regretting it. It’s made me look awful and I’m sure they’re now wondering if I know a damn thing in all the other areas of my work.

Does this happen often…?

..and just think..

we are all having a BAD DAY because some little Chinese 13 year old hacker-wannabe is testing out his new “Hacker’s handbook in the Dreamhost chapter” and hitting Dreamhost’s most vurnable part THE TETHERS! (has anyone ever looked at the firewall logs and tables? Dreamhost’s DEFAULT security rules allow all kinds of security threats and attacks in such as ASSBAY.COM and gives them full access! I had to rewrite the security model a couple times (and DH keep rewriting it back to allow assbay.com and among others in!))

actually, this DDoS doesn’t surprise me at all.. really INSECURE network they have!

I told my acct to backup, and it didn’t and now it’s used my one monthly backup *weeps*

@EJ – Yes. When the data center IS working, it’s very slow. So, it’s either really slow or down – take your pick.

Does this have to do with the Chris-Chan situation?

_____

—–
\ _
\ (_) >

_____

—–
\ _
\ (_) >

My site is down, Help!!!!

my site is still down! anyone have theirs back yet?

still down

I do. Been back up for 10 minutes now

Seems the Secret Service are the only ones happy with DH in here, because DH goes down on them more than prostitutes in Colombia…

All of my sites are still down, because the database server(s) seem to be down. So if I go to a PHP page, it displays a database connection error. It will be a huge pain to switch, but these outages are causing me to look for another host.

Cool, the sites are back!

Time to remove all references, ads and recommendations for Dreamhost. I can’t in good conscious recommend Dreamhost anymore to anyone.

And it’s back up. That was almost 1.5 hours.

at this point there’s no use, so I have issued the master shutdown command to KILL my DH dedi box until further notice..

at least that should protect it from the hackers who are breaking into their systems as we speak!

still nothing!

@giorgos Hmm I was on my DS from 6:30am PST to about 9/10am or so and didn’t experience any downtime with my updates.

Jw

The monitor is Down since : April 28, 2012 4:51 PM CEST

That what I get from monitoring service 24×7 via mail . Still down. So means something like 8 hours downtime!

Site is still down for me. GOGO IRVINE!

How do I setup my account to get an e-mail, text, or phone call when one of my sites is down. It would be nice, if I could send an e-mail to my clients to let them know their site is down and that everything that can be done to get it up is being done. This way I don’t look like a fool when someone calls, e-mails, text, Skypes, or shows up at the door asking why their site is down and I don’t have any heads up. This would be a really nice feature.

My sites are still down. Not fixed yet

April 28th, 2012 at 2:47 pmAngry Designer Says:

@MSCRM – Uptimerobot.com and its free for up to 50 sites

@MSCRM King

That would be a very nice feature, unfortunately, DH never reads these posts.. this place is more of a “rant and bitch box” I recommend that you pop off an email to DH, proper throgh your dashboard (though I doubt those even get read either!).

Mines still dead, too

April 28th, 2012 at 3:02 pmWilliam Xavier Bobberson Says:

I was in Irvine the other day. I had to rock a wicked deuce so I pulled into a local produce shop. Did I have stinging, explosive shits? You be I did. Did they remind me of Dreamhost’s service? Of course! After exiting the dumper without flushing, I proceeded to sex up the “full-figured” cashier.

Oh Irvine, you’ll always hold a place in my heart, especially when I think about painful shits and fat chicks.

Not that my e-mail will be working or anything, but I also can’t stand the crappy service, and I used to love it. I can’t even get this latest site to load in correctly. Lord only knows what they’ve done to the servers this time.

I wonder… will Dreamhost ever get their act together? Probably not.

More like HellHost.

All sites on a ps still down.

All sites still down

My sites are still dead too. =w=

Dito; all dead still…

Now I’m getting a great big happy 403 error. There better be some good compensation for this farce.

I’ve had it with this service, I’m moving to http://www.westhost.com/web-hosting/

It’s 8:30 AM by CET and none of my 20+ sites are working. It continues to read ‘error id: “bad_httpd_conf”‘ and I have lost appx $400 since yesterday. That’s just not fair people, I have trusted you for over 3 years. Who’s gonna patch my financial loss? Are you about to compensate the damage somehow?

I’m also getting the ‘error id: “bad_httpd_conf”‘ on my ssites and haven’t heard back from customer support after sending in my ticket much earlier today.
What gives?

I’m getting only a blanco page. probably can’t reach the DB server

The site has been more down than up in the last 48 hrs

my site is still gone.. 16 HOURS later..

time to ask for ALL my money back…

oops, oh yeah, they took their support system down.. now you CAN’T reach them- or even put in a ticket.. (maybe THAT crashed too?!)

It looks like I will go Rackspace to end with these frequent issues with DreamHost; I don’t want the dream to turn to a nightmare. Thanks for making running ASAP.

Again?

OK. Are you trying to have us leave you in droves? This is very uncool . . .

Website still down with “error id: “bad_httpd_conf” message…

Still down for me.

I am getting this message:

Error establishing a database connection

when I go into my login screen or directly to my site. When I go look at your guys sites it looks like they are up and running????

Are you guys down again this morning? I am not getting any response from support yet.

Oh BOY!

not only is my site still down (over 24 hours later) NOW they are telling me that “theres a DNS mistake” and that I have to “transfer the A records to them”.. I originally had “DNS only” set up.. and what they are asking for now is for me to SURRENDER OWNERSHIP of my domains to them! THAT’S TOTAL BULLSHIT! THAT’S NOT GOING TO HAPPEN! I HAD ENOUGH OF THIS PIECE OF SHIT HOSTING COMPANY!!!!

its one thing to transfer to their DNS servers, but I will NOT transfer the A records to their system.. NOT NEEDED, and, then, like what go daddy does, once I do that THEY’LL OWN THE DOMAINS and they will probably NOT GIVE THEM BACK.. again NOT GOING TO HAPPEN! THAT’S NOT EVEN THE ORIGINAL FUCKING PROBLEM NIGHTMARE HOST!

here’s what I sent to Dreamhost, this is my latest “service ticket” text..


Dreamhost:

First off, unless I’m misunderstanding your “DNS mistake” (not clear at all, even in the wiki) my domains DO point to you.. and ALL THIS WORKED a few days ago..

BUT..

THAT’S NOT THE ORIGINAL PROBLEM!!! I can’t even reach the server ***VIA IP*** (didn’t you see the ping listings?) second of all, and READ THIS CAREFULLY.. I WILL NOT TRANSFER OWNERSHIP OF MY DOMAIN TO DREAMHOST.. (do I need to type that again?)

My security adviser stated to NOT transfer the domain to Dreamhost.. because:

1.. you’ll OWN it.. and then hold it HOSTAGE and THAT’S NOT GOING TO HAPPEN

and 2.. then you’ll CHARGE me MORE $$$ to host the DOMAIN as well as the server (which is currently NOT EXISTENT)!!

WHAT A RACKET!

You have a choice here, either fix the server…

–or–

I DEMAND A FULL REFUND and then I’ll leave this “DreamHost” in the PAST, the choice is yours..

you’re losing 100’s of customers by the droves; that is, until they find out they CAN’T leave because you are holding their domains hostage!

I’m usually very understanding in these situations, but the sheer arrogance and lack of support is just BEWILDERING.. you SAY that you were voted “best business” by California business association?

PROVE IT!

guess I better think twice the next time I work with a business in California! you have NO contact numbers, and your live chat is NEVER online.. NEVER!! so all customers can do is…. NOTHING… send you “service tickets” that probably don’t even get read.. all I have seen are ROBOT REPLIES to my ticket emails…

WARP Radio Network, from what I can tell you are just a crying idiot…… Do you even have any idea what DNS is? You dont have a clue how to set up DNS, dont u?
Your security advisor? LMAO… Go get a life…

It’s a rant for sure from WARP Radio but i tend to agree with him that they need maintain uptime and their customer service needs to be better. I’m still having trouble with my sites and if I was losing money or potential customers because of it, I’d even be more upset. I’ve been dissatisfied with Dreamhost for awhile now. This is especially true when I start looking at other hosting companies and what they offer for the price I’m paying.

This might just be enough to get me to migrate somewhere else.

It’s SO AWESOME to read these statuses and realize that 0 of my sites are down, because I moved away from Dreamhost months ago. :) I urge you to do the same; what’s your peace of mind worth?

yes, it is a rant.. and yes, I know what Domain Name Service is..

My DNS is handled externally.. I don’t need DH to host my domain, I just need them to host a server that the domain points to.. therefore, I used “DNS only” in the configuration. now DH is telling me this is no-longer allowed.

besides, its not the domain that’s the issue, but I can’t reach the server via IP either.. the server itself is GONE.

Sites still down: g2gexchange.com

Most of my sites are still down as well, even though in the panel it says “Our admins have fixed the issue with the Apache services that was causing a ‘bad_httpd_conf error. We appreciate your patience while we got that worked out.”

It’s not fixed. Many of my sites still give bad_httpd_conf error when trying to access them. :(

Dreamhost’s DNS has always sucked. Now MySQL hostnames cannot be created. It shows up in the Use Hostname: dropdown box but not Hostname for this MySQL server:
NO hostnames for these databases… They are inaccessible until you add one!

I really want to see F u guys….. DNS problem is critical. If you require MySQL hostnames to be hosted by Dreamhost DNS, then fucking make it work…. FU!!

I can’t use any of the database, FUCK!!!

Not resolved – I have multiple sites that are all still down – now 24 hours and counting. Submitted a ticket via panel 12 hours ago with no response except the generic panel ‘outage’ notice which wants to assure me “Our admins have fixed the issue…”. Also, no new @dhstatus or blog updates since the 28th…? Longtime customer here who sadly can’t wait to move my hosting elsewhere…

I guess problem has been resolved, for now….

The web server at dreamhost has been unresponsive today…
And ns1.dreamhost.com is having connectivity problem which can kill everything… Google and Level3 DNS was having hard time communicate with ns1.dreamhost.com……. MySQL hostnames took 4 hours to be added…

Looks like name servers are down today for me too.

Time to find some myn and womyn who want to Turkish oil wrestle…

Been with Dreamhost since 2006, and for the first time I’m really, really angry with Dreamhost.

My sites have been down for 24+ hours now. Two tickets submitted, no reply.

The panel outage notice tells me it’s been fixed, with a cheery “Let us know if you have a problem, and we’ll fix it ASAP!”

I love Dreamhost, and I love its funny newsletters, and I love its control panel. But you know what guys? This is it for me. I’ve already registered with Bluehost and I’m moving my sites now.

Are other people still having issues???

Yes. Email and databases for many of my sites are still down.

I am still down and I have multiple sites. This is really too much. Do you not have monitoring in place? How can you not have redundancy? Seriously, guys, this is ridiculous.

AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ? AGAIN ?

People @ DREAMHOST:

DO F*****G SOMETHING!!! YOU DID A SERIOUS BUSINESS DAMAGE TO THOUSANDS OF PEOPLE! 48 hours and still no results?!? It never happened in the world of hosting, my clients will eat me alive!!!

GREBANNIJ DREAMHOST,

PROSTO URODI, GREBANNIE DILETANTI, SAJTI 2 DNJA OFFLAJNA, A ETIM HOTJ BI HUJ.

NAHUJ DILETANTOV!

So, my site was up 30 minutes ago and now it’s down again. And no update? Seriously? I’m not sure why I’m still putting up with this shit. Do we really need to have an outage every single month?

All my sites are still down.. again… and one of my clients has a crucial event they put adverts out everywhere for.. linking to their site… which is down… on one of their most important days… needless to say my clients are starting to get really pissed off, even a little notice when these things are due to happen would be nice, even an email maybe? I only found out when the client contacted me in a panic….

I need to go to Amazon AWS, URGENTLY.

And of course after emailing them regarding all this, there is no reply….

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)
6:52 pm

Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)

Posted (April 27th, 2012 at 6:52 pm PST) by Ralph

On Wednesday May 2nd we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles based data centers.  The fiber ring is being tested to minimize errors and to ensure maximum throughput.  We are giving ourselves a 2 hour window starting at 10pm PDT Wednesday May 2nd.  Service affected will be strictly web content. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup so this should only affect a small subset of customers.  This maintenance should not take the full 2 hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 2nd 7pm PDT: This maintenance window has been cancelled and will be moved to Monday May 7th at 10pm. A new status post will be created to keep track of the outage.

This entry was posted on Friday, April 27th, 2012 at 6:52 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

56 Responses to “Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)”

Wonder how to tell if you’re one of the small subset customers?

Sites down. Anyone else?

Yup, me too.

All my services are down.. looks like I am giving more credits to my customers, with these kinds of credits I am having to give out I am not going to make any money while hosted at dreamhost so time to move

down … about 5 minutes ago. Irvine data center, again.

Is this going to fix these outages that keep happening (like now)?!

dammit

Site down, again! Omg, enough is enough..

gee move everyone out of the Irvine Datacenter since it isnt working properly what a concept

And dreamhost do it again :D

E-commerce website down on a Saturday evening. Perfect!

Dreamhost, you’re the best!

Wow. Again.

Just shared a new bit of content and now you go down. Cheers!

yeah sites down… :8(

Another DDoS ? All my sites down :(

Sites down again. LMAO

April 28th, 2012 at 12:35 pmBarely Credible Hulk Says:

All Of My Sites Went Down in Past 10 Minutes.

site is down. please fix soon!
Thank you!

Site just went down here…

my whole site and server has gone down? fix it…guys…what kind of server management is this?

all my sites are down.
2 customers allready callme.
you cant keep doing this.
at least gime some kind of compensation

I think it is really time for us to move. I really want to give DreamHost a chance but so much downtime for a dedicated server … I am just not sure

guys what is happening? fix this!! this much of downtime

Cyrus I am with you, looking into other packages now

April 28th, 2012 at 12:40 pmAngry Designer Says:

Is everyone tweeting and emailing them? let’s wake then up.

Are there any suggestions for a substitute dedicated hosting away from dreamhost?

My site is down too

All sites down again!!

Too much nickel and dime outages. So many on here say time to move. My question is, is any other host better or do they all perform at this same crappy level? I so want to move, tired of all this downtime. But… can’t tell if the grass is greener anywhere else.

@Bob. I’ll be honest. The problem is the price. I have many sites with VPS.net and WiredTree. They got no issues like this. But you end up paying $60 more a month in comparison to same plans with DreamHost. But they are reliable so you got that going.

All of my sites are down AGAIN. It seems like every single day I have issues with a) sites being down b) slow page loads (longer than 3 seconds) c) MySQL database not responding. EVERY SINGLE DAY!

I’m a Network Engineer for a Fortune 100 company with thousands of servers worldwide and it’s extremely rare to have a server go down, let alone problems like this happening every single day. We have redundancy and fail-over systems in place, evidently Dreamhost doesn’t have that. Heck, I run a bunch of servers at my house and my oldest server (used as a 16 camera security DVR, web server and FTP) has been running 5 straight years without a single issue – and that’s using off the shelf parts.

Outages and sluggish performance is just not normal in a professional environment. The last time I was down, Google ‘punished’ my sites page ranking (which cut traffic 80%) for not being up. It took a week to recover. Looks like I’ll be punished again.

down … about 5 minutes ago. Irvine data center, again.

Site down again!!!!!!!!!!!!!!!

All Mysql is down for me too, *sigh* right in the middle of modifications

YEAH IRVINE AGAIN!!!
F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK F*CK

This is very uncool . . . what is going on? Again???

All of my sites and sub-domains have been down for about 20 minutes. I sent a ticket in, no response.

My two sites are down, again. Using Cloud Flare on both and doesn’t help. This is bad.

With zero compensation. I’m done dreamhost

April 28th, 2012 at 12:55 pmPissed Off Customer Says:

Seems like this has been happening every week. I’m losing $$$. Can someone point me towards a good VPS host. What are the best hosts in order of reliability?

I have only been with dream host for about 4 months. How often does this happen?

Seth @ for me it doesn’t happen often, this is the first in a while

@Seth. For me, it has been happening quite a lot. The server is perfect. The data center thing is messed up.

@Pissed Off Customer, I am starting up a new website, going to be hosted on a Dedicated Box rented from ubiquityservers.com. I’ve been with them for half a year, have had next to no downtime problems or lag issues or anything.

– Alex Penner

@Seth. 7 downtimes for the last 3 months!

I have only noticed 2 or 3 times. Is this a complication from the scheduled work yesterday or some DDoS attack. I am just starting a web develop company and need to find a stable provider before I get too many clients rolling. Who is the most stable?

Are these guys even aware that our servers are not responding? no reply or any response from these guys that they are working on the issue!! this is just plain stupid.

I have a sneaky suspicion that in order to manage costs, DreamHost has reduced its customer support. The chat feature has not worked for me for months. Everytime I click, it says not available. That’s OK. My server has been running perfectly if I don’t consider data center issues. No crash from traffic or anything like that. Just data center issues.

Theres a message on the support page in red

“We have noticed connection issues in our Irvine datacenter. This is resulting in site and database downtime. Our network engineers and system administration team are onsite working to restore connectivity.

You can keep updated at dreamhoststatus.com!”

@ds2264 Check the web panel folks it’s in big red colored letters. Just log a ticket and wait. They haven’t logged anything here yet but it’s been like 8 or 10 minutes so they might not have gotten to the status page yet.

Jw

Hi, we’re a bunch of little old ladies who like to knit.

Google is telling us that our site is re-directing to a malware site, but nothing has changed in the code and WordPress works just fine. There isn’t any malware in the page code itself.

Is this a problem at DH? Someone told me that DH’s DNS may be misconfigured and redirecting people from our site to an advertising site or malware site.

Any help is appreciated.

The “LOLs”

April 30th, 2012 at 2:56 pmrandom customer Says:

@Knitting Ladies this is not DH support and they seldom read these comments. Your issue isn’t related to the post, either. But, so you don’t just hang there…

Your site has been hacked. It’s a problem with insecure themes and plug-ins people have used with WordPress and other commonly used CMSs.

Start by deleting any htaccess files you find (later, you can force WP to create a new one by selecting and saving your permalinks in the dashboard), but you need to do more to keep this from coming back. Seal up the hole that let the hack in.

Read these for help on understanding and cleaning up hacked sites;

http://discussion.dreamhost.com/thread-134262.html

http://wiki.dreamhost.com/Troubleshooting_Hacked_Sites

@Knitting Ladies,

Google is currently listing your site as suspicious, but has found nothing in the last 90 days, I did a scan of your site and found nothing so you should be able to submit a review request @ https://www.google.com/webmasters/sitemaps/ and they will check your site and let you know.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)
7:14 pm

Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)

Posted (April 25th, 2012 at 7:14 pm PST) by Ralph

In our continued efforts to upgrade our network, we will be scheduling maintenance on a piece of our border network equipment on April 25th. The action being performed is a simple software upgrade which will improve resource utilization on the new Juniper router we installed. We anticipate the downtime itself to last no longer than 10 minutes; however, we will be on high alert for the 2 hours preceeding the window as well as during and after the upgrade to make sure there are no complications. We sincerely appreciate your patience and understanding as we strive to improve our service for you.

Update April 25th 10:54pm PDT We’re happy to report that this upgrade is done, and happened without any issues! All service should be working as expected. If you’re seeing problems with your domains, please contact support immediately. — Brian H

Update April 25th 10:21pm PDT First part of upgrade is complete. We are starting the final upgrade now, then one more reboot and we will be upgraded. We will continue to monitor after upgrade is complete.

Update April 25th 10:12pm PDT The reboot of the network equipment is happening right now.

Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

Update April 25th 7:15pm PDT Maintenance is scheduled to start tonight at 10pm. Please keep an eye on this post for updates once the maintenance starts.

This entry was posted on Wednesday, April 25th, 2012 at 7:14 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

55 Responses to “Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)”

April 25th is a Wednesday. :)

Ack! Thanks Peter!

ok, so whats the excuse for the current downtime!

My site is down also Simon ugh!!!

im up

Please elaborate on the nature of the downtime. All services will be down/unreachable/etc?

Down, Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,Again,

Why is my website down YET AGAIN?!

So why, yet again, is my website down? This is getting ridiculous…. And my comment was removed as well.

sites are down.

All sites down here too, I was on errai that was supposed to be migrated

Our website is down as well. Please advise.

How long is it gonna take for the sites to go back up? Could someone give me an update????

This is absolutely ridiculous. My server has been down for well over 24 hours and not a single word from DreamHost. Am I seriously paying for this rubbish?!?! Dreamhost give us an update!!!

You know, they’re telling us about an update that’s not going to happen for another 5 days. Why are people posting in here that their site is down when they should be opening a ticket? There is no global outage right now, go through the normal channels.

On the plus side, thanks for the advanced notice. And it’s nice to see that someone is actually reading these comments as shown by the response from Ralph to Peter’s reply

so, my site has been down for 10 DAYS. Callbacks were blown off, and regular support posts ignored. @dreamhostcare is deaf. I’ve only been a customer since 2000, with about 50 referrals. I just want ti to be back up so I can export my shit out of there.

Patrick, I think you’ll find I do have two open tickets for over 24 hours with zero response from DreamHost and also the deaf @dreamhostcare – I need my sites up and an explanation as to why my whole server and FTP is unaccessible so I’m sorry if you find this a problem. I’ve tried every way to get a response from them and they are just ignoring my issues but happy to take my money!

April 20th, 2012 at 4:07 pmphiya@dreamhost Says:

despite the bold disclaimer, some of us at Dreamhost often do review these comments. Your best bet is to include your domain name in the website field. We’ll do the best we can to help!

The site is down yet again and no way to reach anyone at Dreamhost by phone to find out what’s going on. Site is paid up through August 2, 2012 yet we’re getting bills for non-payment. Anxious to move this elsewhere ASAP but stuck to be able to change the DNS since the domain is there, too.

Point of order: This article is about a planned maintenance for 4/25, where they mention just a brief outage. What does that have to do with other claims of outages, particularly since it doesn’t seem most of them list an actual domain that’s down?

Peace,
Gene

@Marsha Eaton: I can currently find nothing wrong with your site or your webserver, which has a 105 day uptime. It appears your last ticket has already been responded to.

Per phiya@dreamhost.com , my website has been down for 12 days. My server is andromeda, and repeated “contact support” messages, a request for callback, send @ replies to @dreamhostcare etc all ignored. I’ve been a customer since 2000 with many many referrals. Can you please make my site work again?

Speaking as a disinterested third party, Douglas, what site you are referring to? The one to which your signature is linked, http://ugutech.com/, works fine.

Peace,
Gene

Is this problem still happening? I have a customer who is down but I don’t know if it is to do with Dreamhost having any issues. He has lost all of his login details.

well fuck my ass… this is a surprise. No it isn’t.

looking forward to it! :)

My website is giving ‘Error establishing a database connection’ error and when i try to access phpMyAdmin through the panel I also get a server error. Any news about this?

Where does it give that error? When I checked the site linked to your name, it seemed to be OK. I’m curious.

If it’s another site, do you ask DreamHost’s support about it?

Peace,
Gene

My site is hosted in Kigali, where they would supposedly maintenance. 24 hours ago is off line and have not received answers about it. When else will take that mantenimineto to do? Why I can not access the ftp?
On my site, rather than the content itself appears the following legend: “Site Temporarily Unavailable. We apologize for the inconvenience. Please contact the webmaster / tech support to Have Them Immediately rectify this. error id: “bad_httpd_conf”. ”
Please!!! I await your comments … urgently!

@Gene, it does work again now. I have 2 sites with dreamhost and both we not working when I left my post. The server problem must have been short lived. Anyway, happy it is working again.

Why does my site keep faltering? I keep getting CloudFlare errors. This is unacceptable… I’m in the midst of development work, and am losing time to this crap.

@Adam: In my experience, CloudFlare will occasionally display intermittent errors in the first few days as it caches your site. It also has a Development Mode setting where it can be deactivated so you can do work on the site (this is most important for static content). You might want to check the CloudFlare Web configuration options and fine tune them to your needs.

Peace,
Gene

About 80% of the time, if I refresh my webmail, I see either ‘Error establishing a database connection’ of “Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@b5.webmail.dreamhost.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.”

I’ll note that email to webmaster@b5.webmail.dreamhost.com bounces. My domain is glamlabs.com

That first “of” should be “or”. I’ll also note that I don’t have this problem on my home machine, where I’m using a mail client; it’s just webmail.

People, read:
Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

yes, down again for the 5th time in a row within a month. I am getting killed by my emp.

The Irvine Data Center is really starting to piss me off. This month has been nothing but downtime, downtime and more downtime. I am paying you all good money for downtime and its all because of this Irvine DC which IMO SUCKS!!! Will I get credited? Nope… because Irvine is mentally retarded and it would be discriminating against the mentally handicap Data Server if I asked for a break… :-@

Regards,
Very-Unhappy-Customer-Who-Has-Been-With-DH-A-Long-Time-And-Has-Spent-A-Lot-Of-Money-Here!!!!!
Server, DS2032

Thanks DH team, Our sites went down for about 10 mins and are now back up and running. Hopefully this short down time was well worth it, with improvements to finally see the end of the past wobbly moments. Appreciate your working on this, and having the guts to take us down one short more time, to hopefully avoid longer unscheduled down times in the future.

The people here complaining, will hopefully be comforted with smooth sailing for the rest of the year. Until then, I guess its easier to be angry.

Cheers

Jay
Customer since 2006

Site came up then went down

Spoke too soon, back down again… ds1438

Okay, making tea. I hope this gets going soon. I have quite a lot of work to accomplish tonight.

I can access my account & website again!

Back up and tea is ready. Thanks team!

Back up for ds2032. I am willing to take bets on when the next apocalypse happens from Hurricane Irvine though?! Anyone?! LOL

Up and appear to be staying up now….

Still the website is not up? It has been much more than 10minutes. Bring it up. Please!

@ds2032 on the hope this scheduled outage should waiver the odds in favour of stability, put ds1438 down for a worry free remainder of 2012

Disclaimer: We recognise events of 21-12-12 or any impending demise of earth are beyond the control of DH team.

How long is it going to be out guys…..come on do something..!! It is business time for us.

My site is running without any problems. Thanks DH Team. You all did a fine job.

Thank you Dreamhost.

You have responded to me and done right with me. I appreciate it a lot and I am no longer upset. :) Little things like this makes me continue to remain a loyal customer.

Shawn, ds2032

@Jay – LOL, well as long as there is uptime until the end of the world, then I am happy >_<

My site is down…
cPanel shows no files there.

My site has been offline 8 hours as of now, nobody from support has responded to my emails, I have been really nice to Dreamhost about these kinds of things in the past but this is completely unacceptable to me.

I remember being in network customer service and thinking, from time to time, how it was hilarious when a customer blew a gasket over a technical issue. It is pretty hilarious, but it’s not so funny when the guy who blew a gasket leaves you and takes people with him. Those customers, like me, spend money.

That’s how jobs disappear, and when one day you find yourself looking for work because of devil-may-care customer support that you think wasn’t your fault, I hope my tirade here is cracking you up all the way to the front of the EBT line right before you end up out on the street rejected.

Now would you please figure out why my site is offline?

Dreamhost took my advice and added cloudflare. still needs some work but you should be able to keep your sites online with cloudflare. transactions wont be possible but cached pages will work. Good for blogs

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » WordPress 3.3.2 update available for one-click installs.
2:02 pm

WordPress 3.3.2 update available for one-click installs.

Posted (April 20th, 2012 at 2:02 pm PST) by Brandon

WordPress 3.3.2 is now available for new installs and upgrades via our One-Click Installer system.

As this is a security release, it’s a recommended upgrade for all users who run WordPress.

Auto-upgrades have been queued for all customers who have them enabled, but will take some time to process.

You can check if you have automatic upgrades enabled (and upgrade manually if you prefer not to wait) from the One-Click Installer section of the DreamHost Panel:

* Log into the DreamHost Panel, and go to the One-Click Installer Section https://panel.dreamhost.com/?tree=goodies.installer
* Click on “Managed Installed Applications”
* Click on your domain, and a list of your installs will show
* Under the “Upgrade Action” column, you can view and modify the status of auto-upgrades for each install
* Click on the “Upgrade to 3.3.2″ button to the right of the install if you wish to upgrade right away

If you installed WordPress yourself, rather than using our One-Click Installer system, you can upgrade by logging into your WordPress Dashboard and clicking “Please update now” from the alert at the top of the screen.

You can see full release notes at WordPress.org:
http://wordpress.org/news/2012/04/wordpress-3-3-2/

This entry was posted on Friday, April 20th, 2012 at 2:02 pm and is filed under DreamHost Features, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

12 Responses to “WordPress 3.3.2 update available for one-click installs.”

Great! Now how about MediaWiki?

Everything down

Please fix this problem. Dreamhost is managing the server so this isn’t my fault:

Your server is running PHP version 4.4.9 but WordPress 3.2.1 requires at least 5.2.4.

Running WordPress 3.3.2 without problem

Daryl, are you aware that you need to set your server to run PHP 5? Go to Manage Domains in your webpanel and check that you have set that there.

Since updating to WordPress 3.3.2 I seem to be getting a 500 internal server error quite often. I thought it might have been the cache/cookies but I checked it out by doing it at work and at home and deleting the cache/cookies. Never had this issue with the last update. Do I need to change something?

April 24th, 2012 at 5:25 amHappy Dreamhost Customer Says:

@daryl, your site is running not running on Dreamhost, but LunarPages.

I updated and now my leads submitted on Lead form 7 are not being sent.

I just did a fresh one-click install of WordPress, and it installed without the uploads folder. I can’t add any media to the web site.

I tried to manually add the folder several times via web-ftp, but even though I get a message the folder has been added, it doesn’t appear. Media upload in WordPress still doesn’t work.

I can’t find anywhere to change the file permissions in web-ftp.

I appreciate the need for web security, but how can people build their web sites without being able to upload images and stuff??!!!

@ay caramba,
Check the permissions/owner of the one click. Is it your user?
I’ve seen oneclicks not get assigned to the correct user in the past (in which case, uploads will not work, no permissions).

If that happened to you, submit a ticket and DH will fix it.

@Random Emosexual: The specific issue that caused that post is resolved. If you have further issues, please contact DreamHost support via the control panel.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Issue at Irvine DataCenter (RESOLVED)
3:29 pm

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 17th, 2012 at 3:29 pm PST) by Dan J

We are noticing connection issues to our Irvine datacenter. Our network engineers and system administration team are working to restore connectivity.  We will update this post once we have more information.

Update Apr 17, 4:07pm PDT: The issue is now resolved, and lasted for about 20 minutes. Our network engineers and system administrators worked quickly to recover the network, and are continuing to work on any possible residual issues. We will make sure to keep you updated, as well as provide you with details regarding the root cause as soon as we have them. We sincerely appreciate your patience while we continue to get this resolved. -Seohee

Update Apr 17, 5:45pm PDT: We sincerely apologize for the network issues experienced at our Irvine datacenter. This particular incident was a routing table issue and was not associated with the previous outages we experienced at Irvine. This outage was smaller in scope and took down services for 20 minutes. The configuration issue that caused this took down 1 of 4 subnets used in Irvine. The fix is in place now and we will avoid any further problems due to this particular issue. We are continuing to work through significant network improvements and have made this our top priority.

This entry was posted on Tuesday, April 17th, 2012 at 3:29 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

129 Responses to “Network Issue at Irvine DataCenter (RESOLVED)”

There’s no network issue. They’re getting hit with DDOS again. Just switch all of us to different datacenter.

Here we go AGAIN!!!

AGAIN?

AGAIN???????????????????????????????????

hello mate

Really? Again again? Well good luck guys.

#(%*&%*(&%(@#&*$9873247 OK REALLY PISSED OFF NOW

Are you kidding me? Again? Why can’t these connection “issues” happen at 3AM instead of 3PM?

I cannot believe it….

de novo! essa bosta desse servidor Irvine! =/ em menos de 1 mês é a terceira vez que isso acontece! é pedir pra migrar novamente de datacenter =/

AGAIN??? How is Dreamhost going to compensate their customers who rely on the services to make a living and put food on the table that these outages are now costing them money? What is the plan to stop this from happening over and over again? Three times in less than 7 days is unacceptable.

This shit cray.

Hah! This old party again!

FUUUUUUCK SAAAAAAAAAAAAAKE

Already I’m getting inquiries from my readers wondering if I’ve gone out of business. Thanks, NightmareHost.

How many times does this have to happen before you change over the users? This is what, 3rd time with nothing more than a, were sorry?

Here we go again! I just managed to pull my sites back up in google ranking – now it goes down again.

Sweet. I loled.

Seriously, shouldn’t there be automatic backups that reroute traffic when this happens? And what happened to the safeguards that you took us down for 8 hours last Friday for? Weren’t they supposed to prevent this from happening again?

I noticed it 19 minutes and 07 seconds ago when I sent in my support ticket… Irvine is my least favorite place in the world.

ARE YOU ORDERING SOME PIZZA WHILE WE WAIT?

This is getting ridiculous!

this is ridiculous…

Here we go …

What the hell is wrong with Irvine Data Center?????? Guys, please fix this ASAP. It’s like this happens every other day.

I dont understand why they just dont move everyone from the Irvine DC to another DC?

My site came back up but don’t know for how long :)

I want my money back for the rest of the year. This is the worst. Anyone recommend a new host provider?

Cool! I love to waste money! Thanks DH, keep this good job!

fucking loosers
glad i am switching host today

Get ready for the ride!

Hey guys. Agreed you need to set-up better protection against these issues, but also it would be good to be told pro-actively about these issues, rather than having to try and find out what is wrong with our sites.
At least that way we can be prepared for questions from our clients/colleagues when their sites disappear.

It’s up now. Lets see how long it lasts. Looks like this is a regular situation every 3 days.

back down again… so I will post that my customers can use their sites and services for 30 seconds at 5:30 pm CST but keep paying me their money

i’m switching over to Just Host right now.

Why is it always Irvine?

COME ON, GUYS. AGAIN????

Hopefully this is more like the 4/9 issue rather than the 4/12 issue.

I understand that this is unpleasant for DreamHost, too, but this *can’t* keep happening. I doubt any of us can afford to have our sites go down for significant chunks of time three times in one week. You need to do better by your customers, particularly those of us who pay a lot of money for a dedicated machine.

Every host has problems guys. I know DH has bit hit a few times, perhaps too many over the past few weeks. At least they let us know and are on the case to fix it. Hope it works out for all of us. :)

Sigh… down again.

I’m here since it’s free for non-profits. Anyone know of another host that will do a similar deal??

I’ve moved to dreamhost from servage because servage let my site down for 36 hours straight and kept promising every hour that I’d be up in an hour. Apparently I made a fatal bet with DH cause site is down twice a week here since December.

C’mon guys. Is this why you provide hosting on an annual basis and not a monthly basis. That should have been a warning sign…

Dreamhost is turning into LoLhost

This is so fuuuuuucked. The only way these twats will get the message is if we all leave en masse. DDoS my ass – supposedly they installed all kinds of new hardware to reroute if it happened again. Do I smell bullshit? What a bunch of retards – off to Amazon for me. Fuck ‘em.

You guys rock! Would love to know when this shit’s gonna be fixed or… how much discount you will give us because I paid a whole year plus extra ip, certificate and a whole bunch of shit that made money to your pockets. Want my service back!

Everyone, I just transferred by site in 5 minutes over at JustHost.com

The site is now up and running. SO happy right now. Don’t wait anymore, just switch. It’s not hard.

I hate Irvine, CA. Seriously.

After being with Dreamhost for 9 years, I’ve finally had it. All these outages are just too much for me to handle now…especially when I’m sending out my resume and directing people to my portfolio on my web site. Dreamhost is literally preventing me from getting a better job!

I’ll start by moving over my small sites, but a complete move is going to take a long, long time. (…or maybe I should have it hosted on 2 separate companies and then just redirect traffic when one fails. Hmmm…)

Praying my clients aren’t looking for their sites right now. C’mon guys.

So so so disappointed.

Just, way, crazily disappointed

@Stu

Trust me. They always do :) At least, it’s my experience. Everybody seems to know how to use Pingdom these day :)

Enter Serverman:

Nightmare host – goes down the mo-ost.
Taa-ke my domain
throw into oblivion!

Nightmare host – goes down the mo-ost.
Taa-ke my domain
Website than never never loads!

Say your prayers my little websites
Where are you tonight?
Your screen has gone whi-ite!

Dreams of bills, dreams of lies, dreams of server denial
and of competitors soaring above you!

Nightmare host – goes down the mo-ost.
Taa-ke my domain
Website than never never loads!

Thank you thank you…

…and now people are texting me to tell me my site is down. *sigh*

Dreamhost techs don’t look at this post. However, here is the CEO’s LinkedIn profile:

http://www.linkedin.com/in/simonanderson

Maybe he’d like to hear from a few of us…

April 17th, 2012 at 3:48 pmTed Nugent's Hosting Says:

Sorry. My bad. I host Ted’s sites here.

“Just switch all of us to different datacenter.” Amen!

I have been on Dreamhost for about 24 hours. Looks like a just got initiated. I look forward to your future fuck ups.

I think i should moved my site.

It very bad if this happen regularly. It will drop our ranking.

I am moving my website here: http://bit.ly/J9vCYB
I’m already hosting 2 sites for almost the same price of dreamhost and NEVER had any problem! Bye bye DH!

Nightmare Host

I just moved my domain to you guys. Hope you can stop this stuff. Is there any way to move my host server to another data center. If this stuff only happens at the Irvine center, would it be possible to port my sites to one of the other locations? Anyone else have or request this yet? 3rd time in the last week since I moved my site here that this has happened. I have been told repeatedly that you guys at least notify us, but some effort to keep me happy might be needed with my timing luck. Hopefully this will be preventable in the future. Thanks for the effort to fix this…

Honestly. dont’ know I should stick around. wonder if theres other places that are cheaper? And don’t have so many problems. This is dumb!

Just checked, my site came back up.

watch the ride prikface!

Let’s try spamming the same word over and over in the comments. That won’t make us look like childish idiots at all!

This comment section is just like a padded room for users to run in circles screaming. Dreamhost does not ever look at a single comment I am sure. But it may be good entertainment for some random person searching for hosting.

April 17th, 2012 at 3:58 pmLate to the party Says:

I really love how much vitriol comes out in 20 minutes, and then…nothing.

Slackers.

Haza port 22 is open again!

Dreamhost, your site says you have “Enterprise-Level Reliability” and you have a “100% uptime guarantee”. How do I get a refund for all the downtime?

Anyone know?

I quit! I am moving! This sucks! I QUI…oh wait…I’m back up…bye.

We are going to switch now. This is unacceptable. I understand crap happens, but the way this has been handled(or not) is not working. Our site simple cannot be down this much. Since we signed with you several months ago, we have been down multiple days a month. This is 2012, what the heck guys? Not how you run a business.

i have Dedicated Servers server i am move my sites to 1and 1 hosting and i am going rase hell to dreamhost about down time to my customers and what refund for my lost of busness

DreamHost CEO on Twitter = @DreamHostSimon
Ask for a refund for downtime. Their site claims “100% uptime guarantee”. We all know it’s unrealistic, but still…

Hilarious. So glad we switched hosting providers over the weekend. Goodbye HeadacheHost!

I C this is an ongoing issue..

As an online media provider, these 4 days of CRAP has put me OUT OF BUSINESS

Theres NO point of me even to continue, I will now shut my sites down… (assuming I can even GET to them)

What was that link, JustHosting.net ??

well folks, I guess that’s what we all get for trusting a host that’s probably run from some 13 year old kid’s bedroom!

new slogan for DH:

“DreamHost: Internet hosting that’s the stuff of your nightmares!” (black horned stallion NOT included)

-WRN-
:( :(

Yeah.. I’d like to get a refund for all this down time. my huge bill is due this month, or next for a yearly subscrib. should just owe the, 20 bucks i’d be happy.. anyone else know of a cheaper place? besides maybe over seas, where i can’t read, nor understand there language haha

Great… they restored from a snapshot… I was making changes to my site when it went down today. I sync my changes periodically and test. Now it was up and all my changes are gone. Brilliant! Thank goodness I keep a copy of my site on a local system.

Strike the up part… it’s back down.

to all you people demanding a refund, might i direct your attention to the limited liability section of the terms YOU agreed to when you signed up? http://dreamhost.com/terms-of-service/ basically it states, dreamhost is not liable for jack.

as for you people saying you “just” signed up? well then take advantage of the 97day waranty and cancel your account.

its budged shared (mostly) hosting, if you put your mission criticle stuff on dreamhost your a moron and deserv the financial “hardship” that you claim you get when your site goes down. if your making so much freaking money that your life depends on it, maybe you should invest in some rack space some where, i mean, you can afford it if your making THAT much money, right?

Third time down in less than two weeks, I’m moving. Just paid to set up elsewhere, buh bye.

We sincerely appreciate your discounts while we keep patient watching all of this happening.

hey Paul.. eat @ss.

Getting Cloudflare errors and I’m not even using Cloudflare :(

Lol. I was just trying to set up cloudflare and it was completely slow and cloudflare errors happened a lot. Now i’ve turned it off and all my sites are slow.. DOH!!!

I’m confused. I deal with switches all the time. It was a Routing table issue? So someone was playing with the route tables during a unplanned maintenance window? Has this individual been slapped?

ridiculous

that’s what happens when you deal with a middle-man hosting company that is leasing space in someone else’s datacenter. That’s why you get the cheap price and shitty service.

I know there is a lot of complaining and threats on here, but I really am recommending to the 20+ clients I have set up with DreamHost to move their business now. This is just unacceptable, at any price.

Can we get a new ETA? Thanks!

All you guys are too harsh on DreamHost. Yes, their servers are down a lot. Yes, they don’t get back to you with their support on time most of the time. Yes, they have numerous network issues. In their defense, DH is like a kid with down syndrome. They are limited in what they can do. They try their best. You don’t yell at a kid with down syndrome. Leave DH alone and just be patient. Just expect your server to go down soon but accept it as the norm and you’ll be fine.

Why you close issue when port 22 still off? Is not “resolved” for me :-(

Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”

:(

Plese All the websites hosted in dreamhost are down or fallen. Please is the second time that the hosting is down in 12 hours.
This is just unacceptable, at any price.

Dreamhost! WTF! You are costing me money! I thought godaddy hosting was bad….. F this I’m out and never recommending you

Anyone know how to tell which servers are located at this datacenter ?
Is it true (as an earlier comment suggested) that Dreamhost are not actually running their won server-farms / data-centers ?

I currently have no service – and was in the process of an upgrade to VPS. Despite many comments ranting about Dreamhost, I’ve had years of very good service – so this recent spate of DDoS attacks is very worrying as I move to expand my use.

Love reading these comments. I got a new host elsewhere, but some of my sites are still hosted here because its going to be a pain to get them working on the new host… slowly but surely. Also I agree that if you depend on your site for a living, you should not be hosted on a shared hosting service, but something much more reliable. I hate all this downtime as my sites hosted here are down every day. Each day they are down, I send in a support ticket telling them about it and asking for a refund. I finally got a months worth credited WOOPTY DOO!!! At least its something while I keep working on moving all my sites to my new host.

I suggest you all do the same.

Deak, which new host did you move to?

here we go again …. server down guys

I opened up my new Rackspace account earlier this week and am in the process of moving my site over. I don’t know what’s up with DH, but it smells like a disaster waiting to happen.

A few weeks ago I accepted the offer to Dreamhost customer use CloudFlare for one of my websites. As a result of this network issue, that website was unavailable for a while, and Cloudflare began throwing up a big ugly screen saying the website was not available.

After Dreamhost got the server(s) operating again, Cloudflare continued to report that the site was not available. So I turned off Cloudflare in the DH panel. Now Cloudflare says the host is not configured correctly. So I refreshed the DNS info from DH panel. It’s hours later, and Cloudflare is still showing its big ugly “host not configured” message.

WTF??

I’m on nassau server, and I’ve been down for 3 days. Anyone else experiencing this issue? DH has not replied to any of my support tickets.

Welcome to Obama’s America man

> Welcome to Obama’s America man

haha, dumbest thing i’ve heard for ages.

Dudes, it is working now!

Bah is it slow again?

April 18th, 2012 at 7:31 pmFelipe Veigaw Says:

My websites are down! Please fix it
http://www.vejaisso.com

Mine too! In the WordPress front and backend, I get this message:

Error establishing a database connection

The “flint” SQL server appears to be down right now, along with (possibly) homiemail-sub5 — virtually all of my sites are dependent upon having working SQL. I put in a ticket, but I haven’t even gotten an automated response. Naturally this happened right when I was trying to make changes to some pages, so the timing couldn’t have been worse. But no SQL outage is well-timed for my sites anyway… for the majority of my sites, a SQL outage IS a site outage. Please help and get the SQL server rebooted and WORKING, because it’s been crapping out a whole LOT lately and that isn’t exactly good for the sites’ images. Or my image of Dreamhost.

now all my sql servers are down – sites are all down!

Hooray, down again! Server not responding. I guess someone forgot to feed the hamsters.

WTF is up with my email? no response!!!

All of my email addresses are down…not sure what else. They are all business addresses…no email, no business. This SUCKS.

not even a response yet!

sick of this, anyone know a good host

up for 5 mins… now down again.

my email is down, but for some reason i can access it through webmail.
one of my sites is down (same msg as user above. the other two are fine.
ironically the site that is down is a different account from the one on which my email, which is not working, is hosted. help!

My email is down as well. I wish DreamHost would just acknowledge to us that they are working on the problem. No response to my ticker and no updates on this blog. Frustrating.

Hi, can’t get to your support because support contact form is stuck at this point “Please wait while we check the status of your services and fetch test results for your account.”

All my databases look down. Please check and respond as soon as possible (Databases are on “corydoras:hable” MySQL Server)

Look on the bright side… when DH has an issue, they report it to us and they work to fix it. How many here have had to deal with GD? They would not admit they had an issue until you called 5 times and demanded a supervisor, and the first 4 people would tell you that it’s your problem.

@Brownrd I would rather have no problems than have someone to blame for those problems.

Couldn’t even get my support ticket sent.

Panel works fine for me, I reported site slow on some of my sites.

But yeah, sites slow !!! DREAMHOST PLEASE ACKNOWLEDGE SLOWNESS IN YOUR STATUS UPDATES!

Fine then, it never happened.

For the record, further to my question yesterday. My problem had nothing to do with this Irvine outage – a simple Apache configuration error following a server move, fixed by Kristine @ DH support.

Whats wrong with you guys?! My site has been down for three months and I have paid all my dues. I have submitted two support tickets so far only to receive some silly mass emails. Do you want me to switch? Let me know straight away….You can be sure that this will be bad for your business….

Been having e-mail issues all day and after 4 support tickets for different e-mail issues, still no answer.

My phone won’t stop ringing because of DH’s issues. This sucks. I’m quickly losing my patience with DH. It seems this things are every week.

My website down again………………..

I haven’t been able to get my inbox to display in webmail for over an hour now.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency Network Maintenance 4/12 – 6 hour window (RESOLVED)
6:03 pm

Emergency Network Maintenance 4/12 – 6 hour window (RESOLVED)

Posted (April 12th, 2012 at 6:03 pm PST) by Ralph

Tonight we will be doing some hardware changes to boost our network at or Irvine data center. This decision is on the heels of multiple DDoS attacks that have happened today which caused service interruptions. We will be bolstering our border routers by having new hardware installed which will manage our connections to the peering partners. This will allow our BGP sessions to be stable through any possible DDoS that can affect our main border router. As this maintenance starts we will provide you with updates as we make progress.

Update Apr 13, 1:45am PDT: We are resolving this post. Everything is working as expected now with some minor adjustments to be made tomorrow. We will keep you updated if any further major changes are needed in the coming days. Thank you for your continued patience and if you have any downtime issues please contact support.

Update Apr 13, 12:10am PDT: Everything is going great. All dedicated machines were also affected since that data center is our dedicated headquarters. Mysql machines affected below and VPS hosts: (apologies these lists were not put up initially)

MySQL Machines:
banjo, bassoon, cello, fiddle, flute, harp, lute, madal, oboe, octobass, synthesizer, ukulele, viola

VPS Hosts:
homie-vserver252-331

Update Apr 12, 11:50pm PDT: Transition is complete and sites are back up again. Cloudflare domains took slightly longer to come up. Everything is looking great right now. Sorry for lack of affected machines including now:

afzelia, araucaria, ash, cedar, cypress, fir, hemlock, kauri, kaya, larch, miram, mirfak, mizar, mothallah, muliphein, muphrid, muscida, nair-alsaif, naos, nashira, navi, nekkar, nembus, neshmet, nihal, nunki, nusakan, okul, peacock, phact, pine, pollux, porrima, praecipua, procyon, propus, proxima-centauri, pulcherrima, rana, ras-algethi, ras-alhague, ras-thaoum, rasalas, rastaban, redcedar, redwood, regor, regulus, rigel, rijl-al-awwa, rimu, rotanev, ruchba, rukbat, sabik, sadachbia, sadalbari, sadalmelik, sadalsuud, sadatoni, sadira, sadr, saiph, salm, sargas, sarin, sarir, sasin, sceptrum, scheat, scheddi, schedir, segin,seginus, sham, shaula, sheliak, sheratan, sinistra, situla, skat, spica, spruce, sterope-ii, sualocin, subra, sugi, suhail, sulafat, syrma, tabit, talitha-australis, talitha-borealis, tarazet, taygeta, terebellum, thuban, tienkwan, tureis, tyl, unuk, vindemiatrix, wasat, wazn, wezen, whitecedar, yedposterior, yedprior, yildun, zavijava, zosma, zuben-el-akrab, zuben-el-akribi, zubenelgenubi, zubeneschamali

Update Apr 12, 11:35pm PDT: Transition is live right now, service is interrupted. -Ralph

Update Apr 12, 11:10pm PDT: Transition is starting now, please be prepared for possible downtime. -Ralph

Update Apr 12, 10:50pm PDT: We are 20 minutes from making the change to our network. We will be turning on the new hardware and having it take over the management of our peering partner connections. -Ralph

Update Apr 12, 10:36pm PDT: We are near ready to start making our changes live. We will be extending the window 2 more hours just in case. So far we have not had any service interruption due to this maintenance window. -Ralph

Update Apr 12, 9:10pm PDT: Still no movement as far as putting anything live at this time. As soon as we are ready I will update this post. – Ralph

Update Apr 12, 8:40pm PDT: We are not ready to make the change just yet so please stand-by. We are hoping to have some action starting around 9pm. Thank you for your patience. – Ralph

Update Apr 12, 7:23pm PDT: We are about one hour away from testing the changes that we are making. As we progress I will be updating this status post. -Ralph

This entry was posted on Thursday, April 12th, 2012 at 6:03 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

71 Responses to “Emergency Network Maintenance 4/12 – 6 hour window (RESOLVED)”

You guys have some serious work to do, you’re constantly going down. I’ve already started the process of moving to a new host. Sorry.

Damn… so close to first.

Does this mean that the other issue was solved? Will this cause sites to go down? Trying to get into the backend of some of my sites now and they’re down.

I really do hope that this works. We’ll see because 4 hours of extra down time for maintenance is a steep request on top of all that’s happened today. If you say that this is going to improve stability, I say, “We’ll see.”

I guess this is why CloudFlare comes in handy, but I don’t like removing the WWW from my domain.

Hi Dreamhost Team,

I have lost approx. 1000 USD due to the service interruptions today and yesterday.

What about the 100% uptime guarantee? How are you going to compensate for the loss?

Thanks in advance for your answer.

I thought you already were using the latest technology. Apparently it is a waste of structure, only the dedicated server price is high to have a bunch of incompetent and misleading!

I hate to say, this last year has made me question using dreamhosts service for future clients. Not going to put hate on the people trying to solve the problem(s), but it seems from now on I will be looking around for more reliable options.

I’ve been with Dreamhost since 2005 and this is the first time I’ve seen an outage of this magnitude. If they are indeed being DDoS’d, that’s not the fault of the company.

Tyler it is the fault of the company to have hardware not capable of dealing with it.

Hey, don’t tell me this is a ‘previously announced scheduled maintenance’.

Please hurry. I have people looking to employ me based off my website and now they are questioning me.

You’re kidding, right? Sucks!

Hi, I trusted you guys for hosting my startup venture’s website, which is primarily marketed through web portal, and this interruption has caused serious damage to the reputation of my venture. Please respond as to how you are going to compensate for such losses, and are these downtimes apparent in future ?

I am with cypress.dreamhost.com.

Some of my sites were down but they are back up again.

hurry up bros!

Personally, I would like my $120.00 a month dedicated server hosted somewhere else other than the Irvine DC because that DC apparently is problematic. I understand there is down time with all hosting services but when a DC has multiple downtime and you risk losing all these customers, then maybe we need to consider going with a different DC that is actually secure. Just food for though.

P.s. I will not be paying for the downtime I have had today, period.

P.s.s. The largest storm chasing social network online is down, the official forum for the National Skywarn program is down and even a city government website is down…

Down with Irvine!!!

Site was just down again, a good hour before maintenance is supposed to start. My subscription is up in 4 months and I can not imagine I’ll be sticking around with the insanely low quality of hosting I have experienced since I switched to Dreamhost. I came here cause I had a great experience with them several years ago for a friends site(s), so when I was looking it was my first thought. Clearly they have either gotten lazy or greedy because it is nowhere near what it was then.

This is a complete joke. The day I launch my site, this happens. Not good, and I will be going elsewhere. When you expect people to live with the fact that their entire sites and livelihoods can just ignore an entire day of outages during a normal Thursday of a normal week, you are delusional. I will switch my service, as well as post my most horrendous experience with this company everywhere I possibly can. You’ve made a fool of a lot of people Dreamhost. Enjoy the consequences.

Don’t think clouds cant go down too..

Hope whatever your doing works. I know its getting a lot tougher to protect the network as technology advances..

Not sure but looks like they say we have to put in a support ticket.. But we shouldn’t have to it should be a given..
“The DreamHost 100% Uptime Guarantee
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee”

Well I give up for tonight hopefully by tomorrow these people figure out how to do their jobs.

Shouldn’t you guys do this after hours? This is probably the most prime time people are viewing websites. Our auditions went up this week and it’s really important to keep it up. I was responsible for moving the servers to Dreamhost guaranteeing that you guys are very reputable. This is the second time it’s gone down this week and I’m getting a lot of heat for it.

Just another problem. Instead of letting our sites time out. Can’t you at least put up a maintenance page for us when you do this? At least let our users know it is the server’s problem, it does a bit set back on marketing.

Thank you. Hopefully this will fix the problem.

God Bless America and DreamHost.

yeah tanner’s post. do this DH

@Tanner – I have been preaching about that for ever now. They could have some backup server where sites can go temporary and a page shows’s saying something like “Site Name is being updated at this time, we should be back soon!”. I know its not impossible. And I bet a lot of people would B****h and moan too. Instead we get errors and lose users and customers… Come on DreamHost, do some ping pong table and all that crap budget cuts and take care of your damn customers! Because without us… you go down the drain… ;)

YOU KNOW PRIOR TO HAVING A DDOS ATTACK MAYBE YOU GUYS SHOULD HAVE ALREADY UPDATED YOUR FREAKING CORE ROUTERS TO AVOID THESE TYPES OF PROBLEMS? IS IT A SURPRISE THAT YOUR DC IS UNDER A DDOS ATTACK? FUCK NO IT HAPPENS ALL THE TIME, WHAT THE HELL ARE YOU GUYS THINKING? — “WE ARE DREAMHOST, WE WON’T COME UNDER DDOS AND WE GOT OUR FINGERS CROSSED JUST IN CASE?”

its down again

April 12th, 2012 at 6:49 pmI should be on reddit Says:

Has anyone else spent their day here instead of reddit?

@I should be on reddit

In my case, it should be G+

My VPS is die, please support quickly

Our expert technical support representatives are available 24/7 via email, live chat, and you’ve even got the option of reaching a human being over the phone. Our 100% uptime guarantee means we take uptime seriously.

I just joined Dream Host a week ago. I was told that they had a good record. Should I begin to worry? I would really appreciate some input with users that have been with Dream Host for a significant amount of time.

Please list the servers that are effected by this maintenance. I am having issues but without this list I cannot tell if my issues are related to the maintenance or if I need to look elsewhere.

Thanks and GL with the work ahead.

Somebody here know what other hosting can we use as alternative of DreamHost?

They were still blaming me for DDOS attacks for!

April 12th, 2012 at 7:06 pmCloaking pages temporarily Says:

Techxperts,

Is spinning up a domain with a a “We are currently upgrading our servers and will be back online shortly” message and forwarding/cloaking all your domains that are down towards this address with masking a good option?

Thanks!

Instead we have errors and losing users and customers … Slutty big that make agent!

April 12th, 2012 at 7:07 pmCloaking pages temporarily Says:

*I also meant a domain on a different hosting provider (obviously)

Just launched an app in the app store that JUST went live – it relies on getting XML data from my website – and of course my site is hosted at DS. THIS IS BS!

Ah, I see what’s happening here. I was starting to go crazy trying to figure out what I had screwed up on my WP site. I also just activated CloudFlare the other day and was wondering if that was the culprit. However, if it’s a DDOS attack, there’s really nothing to do but wait it out.

In future, it would be helpful to have a banner or something show up on panel.dreamhost.com when you’re experiencing system-wide issues. I didn’t think to check Twitter for about an hour and that’s when I saw this.

Good luck!

Site UP Thank DH hehe

Better late than never — hope it works. Glad I jumped on that CloudFlare offer for my main site last week. I’m grateful for that, as well as for this status update blog — my last webhost (Webhostingpad) was such utter crap, this still seems pretty good to me. (Yes, I know how pathetic that sounds.) Of course, I’m just a web dilletante. I’m sure I’d be freaking out if I were trying to make a living here.

Moved all my sites from Dreamhost 3 months ago, haven’t looked back. Glad I made the decision when I did.

I wsh dreamhost the best but they’re oversubscribed, plain and simple.

its down
..

again
..

lol

If the hackers have any honor they will immediately stop… if there is any honor among thieves and hackers. They have caused great misery and loss of profit for small business and entrepreneurs around the world. Pick on the big guys if you are so tough – but dont pick on a single father who just wants to spend more time with his 4 year old genius little boy.

Why dont you hackers pick on big Pharm? Or how about big oil? Why destroy the liberty of small shop owners? What the hell are you trying to do?!

I second Pins Fast response!

Why do people who have moved off of DH still come here to this site?

Let’s all remember that DDoS attacks can be mitigated, but we can never be completely immune to them.

Look at Visa, Paypal and other web giants that had their sites taken down by hacker group Anonymous via a similar attack.

It’s simply not possible to be completely immune to it. So imho these things ought to be incorporated into your business plans if it is to succeed.

I’m glad to see Dreamhost is nevertheless acting to improve it’s infrastructure, so despite some hours of downtime we will benefit in the long term.

@Don Now I don’t understand that one. I have moved some of my sites though. Still have a bunch that were done for hours today. But hopefully, I can move them in the upcoming months so I won’t have to deal with this again.

April 12th, 2012 at 9:25 pmThanks for updates Says:

Please continue with updates as they are appreciated. I will be watching and considering forwarding my domains to a page with a temporary message in the early AM if this is not resolved.(fingers crossed for you guys)

Best of luck!

I know I’m new to this whole web hosting business and all that, and it’s probably not cool to say this…Thankyou guys for all your hard work.

Prepare for possible downtime???
We have been down for 20 hours now !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Is there a list of servers affected by this operation?

When will this be done? Need to get something published right now!

this is funny…

I’m so glad I’ve moved my sites away from DreamHost. I used to be on the “Yo-Yo” VPS plan at DreamHost for $150 per month, but I now pay less money AND have had ZERO downtime since I left.

It’s fun to check here once in a while to see all the cr@p I’m missing!

:D

As suggested by a few, we can have some message saying “Site is undergoing maintenance” instead of losing our customers. Thank God I dont have my e-commerce site hosted with dreamhost.com. Already I am being whacked for the downtime..

We should call the Irvine DC….nickname Hurricane Irvine since it is the cause of all this mess lol.

My sites are still down!!!!!

What the heck!!

My sites are down.
Lots of downtime happens + Passwords hacked + Eval injections and no direct assistance.

You should do better than this!

glad to see you guys are working on resolving this issue. look forward to it being finished

Sites are all up. Thanks team

It is not clear if Dreamhost thechnicians, is a complete idiots, or a big asshole!

I’m on afzeila and getting constant “Internal Server Error 500″ from customers visiting my site….never happened until yesterday…still happening today. any ideas whats up???

Sites are not all up. My site is down and I’m getting “Internal Server Error 500″ messages too. AND no one from Dreamhost support will reply to my pleas for help – saying you’ll respond “within 24 HOURS” is a cop-out.

Looks like my sites are up. Hopefully this is the end of the flakiness.

Down again? Really?

Please somebody tell me is Gator Hosting better? Less outages? This is ridiculous! 2nd day now? I am losing time and money!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Issue at Irvine DataCenter (RESOLVED)
3:46 pm

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 12th, 2012 at 3:46 pm PST) by Oscar

We are noticing intermittent unavailability to our Irvine datacenter again. This is due to another Distributed Denial of Service (DDoS) attack. Our network engineers and system administration team are working on mitigating this attack.  We will update this post once we have more information.

Update Apr 13, 1:54am PDT: Our maintenance window has concluded successfully and all services are restored. If we have any further plans we will let you know in a new status post. Thank you for your patience.

Update Apr 12, 6:10pm PDT: We’re proceeding with an upgrade to our infrastructure which should not only alleviate the current issue but also preclude us from suffering similar issues in the future. Please refer to our new status post with details on that action. We will update this post again once that has completed and we have had a chance to access the results. — Brian H

Update Apr 12, 4:45pm PDT: We are still experiencing unstable service as a result of the DDoS. We are working with a peering partner so that we can mitigate any further issues. Our live monitoring of the incident continues and we are able to get the flaps (temporary disruption of service) mitigated as much as we can. Our efforts continue till we get through this. –Brian H

Update Apr 12, 4:10pm PDT: Multiple DDoS attacks have caused our service to become unstable again. Our security team noticed the attack and where able to deflect and stop many of them. We were able to then identify and stop another retaliation attack, but that did cause some downtime you just saw.

The changes explained in our previous outage post are still scheduled to occur and will ultimately help if this situation comes up again. We will continue to work on the problems and stay vigilant to ensure that these attacks cause no further problems. We will have further information in our maintenance status post. We sincerely apologize for the problems caused by these attacks today.

This entry was posted on Thursday, April 12th, 2012 at 3:46 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

152 Responses to “Network Issue at Irvine DataCenter (RESOLVED)”

YOU SUCK DREAMHOST

Ok, so, if this is actually a DDoS attack, how does dreamhost suck?

Seriously…

why a new report post?

I don’t get it. My old hosting service never seemed to get attacked. Why does DH constantly get attacked? Can someone explain this to me?

does it mean many sites are being affected ? like mine? i cant load my site for 30 minutes now.

James, you suck too

unbelievable thats 3 outages in the last day

Glad they reposted this as our site is down–again after only being up again for a couple of hours. At least they know it’s a problem. We’ve lost thousands of dollars today due to this. Sigh.

OMG! From where come these attacks?

The sites I have on a cheap DH shared server have been fine all day. My VPS has gone down more than Jenna Jameson… Why?

All I have to say is lol :) 100% up-time guarantee?

Thanks Joe!

and time to resubmit a new rport ticket for these days as well

Susy, I’d guess they’re not getting attacked…but one of the sites they host IS getting attacked. When that happens, we all suffer. I wish they had lots of smaller data centers instead of fewer big ones. At least if someone got attacked then, it wouldn’t affect so many people.

@Brad Shared hosting sites are on the expensive servers. It hardly goes down. When you upgrade to dedicated server, they put you on the cheapest.

@Cyrus – I e-mailed them last time the site went down to get compensated. They just ignored that support ticket.

Well then lets just make a NEW status about the SAME FUCKING ISSUE.

#dreamhostsucks lets make it a trend on twitter.

Sites up.. again. for now.

My site just came back online

Dear DreamHost,

I know you guys are doing all you can. DDoS attacks are a pain to deal with. Get well soon :P

@Gnanes. I am really sorry. I don’t mean to make fun. But instead of crying about it, I make joke about it. In reality, the refund won’t help you. If this continues, your search rankings will get affected. You will lose readers. Your site will leave a terrible first impression on new visitors. Your credibility as a business will be affected. When I say you, I mean people who host on DreamHost (which includes me too).

The fact is, the refund is useless. It really is. It won’t solve anything. Even if they pay me my $1500, which is about what I paid them last year, we lost much more than that. Also, when we complained about downtimes, we were asked to move our sites from DreamHost by a certain date.

My site just came back online [2]

“April 12th, 2012 at 3:48 pmPatrickD Says:
This Celso character seems to admit it’s his fault here: http://www.dreamhoststatus.com/2012/04/12/network-issue-at-irvine-datacenter/#comment-273812

I think it just means his hosting had been compromised as part of an attack. Pitchforks away.

Down again, this thing goes down more than a cheap whore.

Site on, site off, site on, site off.

Do you guys not understand what a DDoS attack is? You can’t just press a button and it fixes it. I’m sure legal action will be taken against the attackers.

@Cyrus – I was joking. I know it sucks.

I e-mailed them asking my sites be moved to a different data center. Everyone else should do that as well. Same datacenter went down couple of days ago because of DDOS attack. Hackers must be targeting that datacenter for a specific site.

I don’t care if legal action is taken or not. My sites are down all day

Guess I’ll be spending my weekend migrating to ANYWHEREBUTHERE.com

Dear Dreamhost….

I want you to compensate everyone for this outage. We should all get a free month of service.

And if you don’t think that’s fair I think we should all launch a campaign to let everyone else know how we feel about this service.

again and again and again and again

Free month of service yea maybe on another companies hosting plan sure but i honestly don’t want anymore time dealing with this shitbrain company.

@Gnanes I hadn’t even considered doing that, the last couple of incidents at this data centre didn’t affect me this one has, im intrigued why contingency isn’t in place for such challenges, surely the other data centres should come online and distribute the load across the rest of the network to keep the service running properly

I know GoDaddy blows. What are other good options? NameCheap? What else? Squarespace with offer code nsfw4 will give you something like 10% off.

“Comments posted here may not be viewed by DreamHost staff at all.”

Time to move on – these comments are just a ploy so we get mad at each other and/or just amuse ourselves to get us to temporarily forget our real problems…

They should give you a free month so you can move. Not to stay. I already have accounts with two other companies. I probably can’t afford to move my current server with DreamHost. But I’ll suck it up and move. I am not out of my mind enough to stay after the hack breach a couple of months ago and now this. Now keep in mind, the DreamHost people promise things but they never live up to their promise. They say they will give credit but never given us any so far as I can tell

You guys just cannot afford the amount of users you got on your back.

Do not believe it is my account That would overturn the DH They are under attack and want to blame someone it is obvious

WiredTree, VPS.net, LiquidWeb. Check them out if you need a new host. At least, those guys have not let us down. We have over 200 sites, 150 of them on other hosts than DreamHost. :)

All of my sites are back up.

AGAIN 4:14:59PM

Down again!

These things happen, it’s not been down that much really today – all things considered.

My site is down.

site is up, but very very very very very slow

Mine has been up for 15 minutes and operating normally

@Anthony how long has your sites been down? I’d like to know when you make general statements like that. My sites have been down for 5 hours today on a dedicated FULL MOON server. They were down earlier this week too. This stuff happens but I don’t remember having this much downtime.

My site is up again

Up & down, up & down, finally at 4:08 up again. This reminds me of a guy named Mike I dated – once!

Larry – lay off Celso. One script on one site could not produce a DDOS attack, it takes thousands of infected servers from different sources to execute an attack that could swamp a host the size of Dreamhost.

That said, I fault Dreamhost’s Control panel for making hacking a Dreamhost site pretty easy. When creating a domain in the CP it offers to create the FTP password for you and then creates an aLpHa only password. The next day Dreamhost publishes all the new domains it got and viola, breaking an 8 character alpha only password is child’s play. That is probably how Celso, and another hundred domains a day at Dreamhost, got infected. Dreamhost needs to remove or re-tool that “feature”. Too many site owners are not going to realize the vulnerability and change the password to a hard password. But, in Celso’s defense, a DDOS would originate from outside of the datacenter, not from inside. An inside job could be isolated and stopped almost immediately.

and down again,

Too bad I can’t code at a mile a minute or I might actually finish setting up my site in the few seconds of uptime i get.

I agree with you Sally

steve you may want to look into serverpress its an excellent tool especially for instances like this.

oh and sites are back up again

I’m still threatened BY Oscars?

my site is still down :(

How a company like DH such a thing happens?

“Larry – lay off Celso. One script on one site could not produce a DDOS attack, it takes thousands of infected servers from different sources to execute an attack that could swamp a host the size of Dreamhost. ”

From the start of your comment, it would appear you had a sense of humor. ie. about the man named mike.

Lighten up. Which is what im trying to make of this @)!#!@$ situation.

Up for a while, but now down again.

Larry We are all @)!#!@$ with DH

ps61126 on homie-vserver263 is down once again…

trying to catch up on today’s lost work when…the fekking thing goes down for the 20th time.
(╯°□°)╯︵ ┻━┻)

AGAIN? AGAIN? AGAIN?

Really gotta lay off Dreamhost. Sh*t happens. One can only hope that chaotic running around is taking place to get us back up and running.
Its almost 8pm here. I shut down at 6. Waiting. waiting. Got to get this clients site online if I want paid! Come on dreamhost, show me the money!

Lol. We are down again. haha. Hilarious

They’re playing ping pong.

We’re still down. I guess I should turn off add-campaigns driving to our site, no? I do know that DOS attacks have seriously spiked everywhere lately. Instead of attacking DreamHost, guys, how about we use our collective brainpower to help . . .

Grin. Orrrrr we can sit back in our over used chairs, where the padding has fanished, kick our feet up, light up a cancer stick, open a beer instead of the mountain dew in the fridge and work in-between up and down spurts.

fanished? vanished!

so is this why i cant access my site thru my FTP? and my site was up now its down. down and up. weeeee!!!!!!!!!

I am getting good at working in 2 minute chunks. That’s the total amount of time I have access to our site all day.

@Wall yeller: aye. like wise.

I know everyone has noticed this:

“Comments posted here may not be viewed by DreamHost staff at all. ”

but at the moment, I just find it funny lol. It’s not like the DreamHost team can fix this mess even if they read this or your e-mails. lol. I mean after 6 hours downtime today or however long it has been (since 11 AM EST), can this get any worse?

April 12th, 2012 at 4:54 pmThe Credible Hulk Says:

So I get a call from a potential Client today:

“Saw your email and really like the work you did, have any other examples?”

Me: “Err uhh, yes lots of them but I am performing upgrade maintenance at “My data center” so let me email you later tonight”

haha

@The Credible Hulk

that’s how you do it.

I Still like you! keep it with the hard work.

@The Credible Hulk

the mob’s new.. umm eaa.. ting .. DDoS attacks

HEY DON’T WORRY GUYS I FOUND AN EQUIVALENT TO DREAMHOST!!!!!!

http://www.nouptime.com/

I apologize Larry. I didn’t mean for that to be as harsh as it was. I too am taking this lightly. I just noticed Celso being mentioned a bit on this and the other thread and wanted to assure him that this is not his fault.

I’ve been running sites for 16 years on everything from Rackspace to a couple of servers in the back of Joe’s Radiator Repair. This kind of thing happens almost everywhere. Bigger hosts like Dreamhost attract the hardest, meanest attacks, whereas I never had a hiccup at Joe’s Radiator Repair…. This down time was probably the longest I’ve seen, which means it was probably the most sophisticated attack I’ve been taken down by.

can’t even get to youarethescene.net or http://www.yats.cd booo!! i’m so pissed!

April 12th, 2012 at 5:02 pmThe Credible Hulk Says:

@Ryan – I’ve migrated all of my sites and noticed an amazing increase in the amount of uptime. I just installed an AOL trial cd and i’m moving at blazing speeds.

moving all my stuff off dreamhost, please tell people about this service of dreamhost

DH is very amateur

Well. It has dawned on me. In between working in quick spasmatic spurts when the servers go up, Ive fubared most of my work as some uploads made it, some didnt. Some kinda made it, others… ehh not so much. I should just stop before someone gets hurt.

Aaaand … still down.
I’ve lost a day of sales.

treyvonn martin give me strennnnngth

Rarity nodded, feigning concern, “Well, Mule darling, I suppose it’s not normal, per se, but normal is for the common sort. Rarity has always demanded the exceptional. You simply haven’t read the correct books. Fear not, my little Mule, you can borrow them. I’m sure you’ll enjoy them quite as much as I do.” Rarity was glad that, for all its unsophistication, the Mule could at least read well.

The Mule seemed only partially convinced. It spoke again, “Well, okay, I guess. But… um… about when we get… intimate. Could you stop calling me a ‘dirty disgusting Mule’ and raving about how much I sicken you? It sort of hurts my feelings a bit, sometimes. And I’m not sure why you want me to bray when you spank me, or why you spank me so often, for that matter, but it’s really weird.”

Rarity tilted her head. The Mule had never objected to that sort of thing before; The Mule seemed to enjoy it all as much as she did. “But darling, it adds so much spice. Surely you don’t want lovemaking to get boring, do you? Besides, we both know what being spoken down to does to you, and that certainly helps things along, and a little discipline is such delightful foreplay. You certainly didn’t seem to mind it last night.” The Mule blushed at the reminder of its usual tastes.

“Alright, I guess that makes sense. But could you at least stop calling me Mule? It’s a bit offensive. And possibly racist, I’m not really sure. You used to call me by my real name.” The Mule looked up at Rarity pleadingly.

The dressmaker clicked her tongue in annoyance, “Fine. I will occasionally call you by your name. But you must remember to address me properly from now on. In any case, I grow weary of this conversation,” Rarity extended a hoof out toward her Mule, “You may now lick my hoof clean.”

@Kiri: Bwahahhaha

Can we figure out who keeps launching DOS attacks against Dreamhost?

Im going to go install Warcraft 2 and relive my carefree youth. Ill check back in later =p

Asian people

It’s probably the lord Jesus himself. I mean these guys are so incompetent … the lord is showing his power with DDOS attacks. Now that BS aside, I keep wondering how this day could get worse, and it does keep getting worse :) I have not had 1 stable hours since 11 AM today. It’s now 8:15 PM. Dedicated server my butt

DOWN AGAIN :> GIVE ME MORE!!!!!!!!!!!!!!!!!!

For all these complaints about outages, I really think the important thing to look at is the source of these DDOS attacks. It’d be one thing if Dreamhost screwed up and burned out hardware, but this is an external force disrupting usage.

A DDoS attack is a very difficult thing to combat (see learn more below). They are intended to harm a high-profile hosting company or Website by blocking service in the hopes that their customers will blame them for the outage and cancel their accounts.

Hey, I’m frustrated too, but Dreamhost has been great for me for years. Hang in with them for this one very bad day!

Learn more: http://en.wikipedia.org/wiki/Denial-of-service_attack

ET! ET! ET!

Peace,
Gene

New problem. My server is available from Houston http://network-tools.com/default.asp and Paris http://ping.eu/traceroute/ but CenturyLink (Qwest) routers can not access it.

This one very bad day has come at the worst possible time and I am not being melodramatic. We are pushing a whole new site from dev to production around an event that starts tomorrow and goes through Sunday. Now faced with having the old site (if anything) versus downtime for the push. Either way, we are screwed. Couldn’t have been a worse time. Can they identify what is being targeted? Is it a site within their network or DH themselves? Plus they say somewhere that they haven’t installed the same level of protection in Irvine as in LA? Um, why not? Where is the CEO during all of this? Where are priorities? No automatic redundancy as some extra cost on their end so as not to lose customers (whether their “fault” or not) when this happens. I don’t care who is at fault, all I know is who is responsible to me and that is the company that I pay so as not to have to deal with this.

@Freyar can I tell you a secret? DreamHost does not give a rat’s behind about security. It’s very well known in web hosting forums and this industry. DreamHost is about offering budget plans. Their servers when they work give you great bang for your buck.Unlimited bandwidth, super cheap hosting plans (even dedicated), all kinds of extra goodies ….

But this is not the first time DreamHost has dropped the ball. We all haven’t forgotten the FTP leak issue from a couple of months ago or so.

To say DreamHost is not at fault at all is just absurd. Sure. The DDOS attacks are nasty. But it’s not hard to attack DreamHost. They really don’t offer a whole lot of security. Use Nginx? You are even more screwed (unless you know your stuff). Now I understand many DreamHost customers are happy with what they get. Great for you. Our experience with them as a blog network has been abysmal.

I am pulling for DreamHost. I hope they fix this soon for the sake of all of us. In our case, we would like to have a couple of stable days so we can prepare for our move.

Jose – I’m guessing this is not being directed at the datacenter itself. I can access my sites from places like Houston TX and Paris Fr, but Centurylink & SBC points in US and Australia can not access it. And that changes from time to time throughout the day. It looks like this is an attack upstream from the datacenter itself.

What is the most viable option for keeping sites up during downtime with any hosting provider?

Cloudflare seems to suck and it just has a nicer message saying my sites are down with cloudflare branding.

Will a CDN service fix this or help? any advice is appreciated.

Thanks!

Once CLoudFlare is up for a few days, it should cache enough of your sites to keep them running in case an outage occurs, at least in terms of static content.

Peace,
Gene

Here’s your solution DH… http://www.prolexic.com/

SIGN UP NOW!

@Gene – Thanks for the response. I added one site after the last outage and I suppose it hasnt been long enough.

@Cyrus If your business is that critical, then why are you going for a “cheap” option? Why not pony up the cash like the big-boys do? You get what you pay for. For a smaller site like mine, I deal with this as it comes.

I know of the leak, and frankly it is what it is. Budget Plan = Budget Service and I still hold that the instigators of this DDOS are the ones responsible for your outage, not Dreamhost.

@Freyar Well, that’s why we moved 150 of our sites and moving the rest this year. Also, please note we are on the third most expensive DreamHost dedicated server. We also use plenty of security solutions to keep the server secure. But we can’t do anything when DreamHost is attacked like this.

Trust me. We pay like the big boys. But we have so many server plans, we have to go with a cheaper plan for less important sites. We have accounts with VPS.net and 2 other hosts.

A budget plan is when you pay $6 a month. Or maybe $60 a month. But we pay way more than that. It is budget in comparison to Rackspace. Yes. We don’t pay $750 a month like MediaTemple charges here but in total we pay more than that.

Also, this is a shameful PR that shows what the industry is all about: http://www.timesunion.com/business/press-releases/article/DreamHost-Awarded-April-Host-of-the-Month-by-3477000.php

A bunch of affiliates pushing hosting solutions they have never ever tried.

Just curious, does anyone have a rough idea of how long it usually takes Dreamhost to sort this sort of thing out (or how long before the attackers get bored)?

I would like to know that myself Sean

Did they all go home for the night? It would be nice if they updated regularly at least.

Damm!! I am brand new to this hosting service and in 2 weeks my site has been down 3 times during peak periods!!! LAME, bad first impression Dream Host!

FUCKING SHIT LOT DOWN MERDA DE SERVIDOR, hora que voltar tiro o BD dessa bosta!

I agree, still have a problem on my site…It was temporarily up about 10 mins ago. now its permanently down. I live in Australia, so i thought i would go have a sleep. To see this is still happening when i woke up. It was frustrating.

mine isn’t up Celso

Site Down :( hahahaha

STILL DOWN! Thats over 8 hours down time today.. way more than 97%

I was happy not to over hahaha :(

can’t get any work done today for more than 5 minute increments :/

Yup. My sites came up for 30 min but now completely down

this is so embarassing, I did a tv show to advertise my site, which is on DEDICATED server of course and it does not f**kin work. and it is every week the same. how terrible can this be.

Hi Dreamhost Team,
I have lost approx. 1000 USD due to the service interruptions today and yesterday.
What about the 100% uptime guarantee? How are you going to compensate for the loss?
Thanks in advance for your answer.

Yup. Sites are down. Awesome.

Fingers crossed that this gets figured out soon… (7:15pm MT)

Site is UP :) Down :( UP :) Down :( UP :) Down :( UP :) Down :(

Is UP now :) irraaaaa1

UP .. DOWN … UP … DOWN … YAY

Is DOWN hahaha :(

now its down

anyone know how to get a little picture next to you website name?

gman are you talking about a favicon ?

its up!!!!! go go go go go

Gman down hahahaha please wait :)

wth. 10 mins on, 10 mins off.. terrible

I think everyone with a dedicated need a one month free hosting at least. I`ve also lost a lot of clients and income because of this.

10 mins on :), 10 mins off :( .. terrible hahaha

They gave me a $3.00 credit for the downtime, haha!!

gman still down.. yeah fivia ka?.. i’ll look it up. site still down.. THE WORLD IS ENDING TODAY!!!!!! RAAAAAA!!!!!!!!!! WEEEE!!!!!!!

i have to go to work. 80(

everyone cheer as loud as you can when the site is up. I’ll hear it while im working outside tonight.

BYE EVERyonE!!

ITS UP!!!!!!!!~~~~~1111111!!!!! GO GO GO GO GO OG

I’m glad to hear that you’ve upgraded your infrastructure to deal with future DDos Attacks. Those are really bad ones and I know my stuff… I sincerely hope that your upgrade will do the trick to prevent some serious DDos attack from working.

It is not clear if Dreamhost thechnicians, is a complete idiots, or a big asshole!

Thanks god! Yesterday we finished moving our last site out of dreamhost :) We werent able to stay for more than two months, its nice they will be returing us the money of the two year prepaid plan we bougth.

DDoS attacks are hard to stop no matter what host you are with. I had one happen on Monday with Rackspace. They said they could only recommend to use Cloud Flair other than that, they can’t stop or prevent DDoS attacks. You block one IP and another instantly pops up to continue the attack.

I can’t believe so many people on here angry and blaming DH… I would love to see your lazy butt stop the DDoS attack.

Dreamhost thechnicians is a complete idiots, or a big asshole?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network Issue at Irvine DataCenter (RESOLVED)
9:22 am

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 12th, 2012 at 9:22 am PST) by brians

We are currently experiencing issues with our Irvine datacenter. We have our network team looking into it and are working to get service restored as soon as possible. As soon as we have further information we will be updating this post.

Update Apr 12, 3:25pm PDT: We are confident that this incident is resolved now.  Distributed Denial of Service (DDoS) attacks triggered a instability of BGP routes to our peering partner.  Now that we understand the source of the outage we have installed a DDoS mitigation appliance to address future DDoS attacks.  We will also be announcing some further maintenance to our routing infrastructure which will compliment the current hardware and assume the role of managing our BGP routes.  The addition of the DDoS solution will help prevent near term issues, but our commitment is to upgrade our networking infrastructure to provide you the stability you expect.  Please keep an eye on this status post for future maintenance announcement.  – Oscar

Update Apr 12, 1:30pm PDT: The routes continue to be stable at this point (just to note, this issue only impacted one of our three data centers) and we are seeing a decrease in reported outages as we are vetting your customer service requests. We have also been spot-checking sites reported as down (both from earlier requests and current ones) and so far they are all coming back as up and functioning properly.  We are still monitoring for stability and will continue to update this post until we are 100% certain that this issue is resolved.  – Oscar

Update Apr 12, 12:32pm PDT: The peering connections have stabilized, all routes restored, we will continue to work to ensure stability and will manually reset connections if there are any further disconnects. We will continue to update this status post until further notice and we are 100% assured of stability. – Ralph

Update Apr 12, 12:00pm PDT: We continue to see disconnects but we are making ground with some prefixes now working correctly. We are making progress with resets. We thank you for your continued patience as we continue to work through this incident. – Ralph

Update Apr 12, 11:40am PDT: We continue to see periodic disconnects to our peering partners causing intermittent downtime to services in the Irvine data center.  As we see these “flaps” we will manually reset the connections and services will be restored. – Oscar

Update Apr 12, 11:22am PDT:  We are continuing to see the same flapping of our network connections in our Irvine datacenter. Our Network Engineers are still working diligently to resolve the issue completely. – Oscar

Update Apr 12, 11:09am PDT:  We are still working on making some adjustments to improve the situation. There has been some progress made but our Network Engineers are continuing to work towards a solution. – Oscar

Update Apr 12, 10:42am PDT:  Connectivity to our Irvine datacenter is continuing to have intermittent issues. The flapping (up and down) of links is causing high CPU on our routers as well as that of our upstream providers so we’re endeavoring to move some routes manually on our end to work around the issue and restore service fully.  – Oscar

Update Apr 12, 10:22am PDT:  We are still seeing intermittent unavailability to our Irvine datacenter. Our entire team is continuing to monitor the situation and work towards full resolution with our upstream providers.  – Oscar

Update Apr 12, 10:02am PDT:  Our network engineers are on the phone with one of our upstream providers. We are working on both ends and are troubleshooting and some connectivity has been restored. We are hopeful that we are out of the woods but are continuing to verify services in Irvine.  -Oscar

Update Apr 12, 9:42am PDT:  Our system administration team and networking team are still working on the issue. At this point we’ve verified that there is severe connectivity degradation in our Irvine datacenter. As soon as we have further information we will update this post. -Oscar

This entry was posted on Thursday, April 12th, 2012 at 9:22 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

667 Responses to “Network Issue at Irvine DataCenter (RESOLVED)”

Second time. WTF!

WOW IM DOWN AGAIN AT PRIME TIME…I REALLY DON’T UNDERSTAND THISIS THE SECOND TIME THIS WEEK

fuckkkkkkk 2nd time in 3 days

This is getting ridiculous

Unbelievable. Monday and Thursday. Hope you get it back up soon!

god damnit this problem happened 4 days ago and I am paying $250 bucks a month for a dedicated server. GET THIS FIXED …

You were online long enough to bill me this morning …

I love CloudFLare; keeps my website and news feed online for my customers :) Seriously guys; this is the 3rd time in a week; please fix it this time!

What the hell!!. Second time in this week.

Really, why are we still working with this datacenter? This is getting boring.

Life’s too short to deal with DreamHost.

Seriously guys, I don’t know how much more of this we can take. I’ve got majorly expensive marketing campaigns directing large amounts of traffic to my site right this very moment… and the site is down, AGAIN.

3rd time in a week.
WTF!!!!!!!!!!!!!!!

All these complaints, and never any fixes.

Honestly this is pathetic. Just destroy that datacenter xD

Great… third one this month…

Here we go again. Non stop action at Dreamhost.

Please help!
My site http://www.seguroviagem.srv.br is down again, crucial applications running on this domain.
Please help! People are dying here!

DDOS Attack again?

The SAME datacenter as last week? At primetime? While we have breaking news and event coverage. Awesome, Dreamhost. Think it’s time to move on.

What the hell!!. is the Second time in this week. don’t believe this? my site is down!!!!! hELPPPPPPPP

AGAIN this week.!⁻!*”·$

I am broke for the 3rd time in a row. Please do something ASAP!!

This was allowable when I was on shared server cause the fixes were quick and the traffic low. Now I’m on VPS with my employer and this is a high-visibilty fail.
Please fix soon.

Come on guys !! 2 Downs for 3 days with same problem. Are you fucking kidding? People are losing money and reputation. You don’t care about yours but we care about us. How you expect people to stay with you? The worst technical management ever !!!

Is there a way to tell if I am being effected by this outage? I assume I am because my site is down, but I want to confirm it.

Thank you.

As if I was paying you to keep my site offline instead of the other way round…

That was 3 days since the last outage, all logic is we will see it fixed quicker this time and it’ll be a few more days before the next outage. Never 2 without 3…

Be transparent and let us know what is really going on. You may retain a few more customers.

@Neonrelay: There is an alternate theory which states that this has already happened.

You guys need to fire Irvine he keeps messing up your network

Major email campaign launching in 30mins and now down for the 2nd time in a week. How can I move my sites to the server where this status page is hosted? It never goes down…

Just in the process of porting to you guys. Cheaper than my old host, but I cannot afford down time. I was trying to update my page and lost 1 hour worth of work on the page. Should I finish this move?

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?

Ironically this only happens when I have a big project I have to move for a client! hahah… <– nervous laughter

i just launch my website! i cant believe this… how often is this going to happen????

Are you kidding me? Twice in one week. This is actually unbelievable.

hahahahaha this is almost comical how this happens except its real money being tossed in the garbage.

seriously what in the hell is going on.

THIS IS THE E-MAIL I RECEIVED LAST TIME:

Please accept our apologies for being unable to send you a personal support response. All services should be functioning normally again.

We experienced networking issues for some customers for around forty-five minutes starting around 8am that was caused by a DDoS attack to a specific server in the Irvine datacenter. This caused connectivity issues that persisted even after the attack was blocked, and our network and admin team fixed them up soon afterwards. You can see the progress of the outage in the updates on the status blog:

http://www.dreamhoststatus.com/2012/04/09/irvine-datacenter-connection-issues/

We have protect against such attacks in our downtown Los Angeles datacenter and plans are already underway for the same protections in Irvine, but they are not yet complete. Our network admins will be discussing plans to get this done as soon as we possibly can.

Please accept our apologies for not directly replying to your support query. If you have any specific questions or concerns which haven’t been addressed, please contact support from within the panel or reply to this message.

Thanks,
Justin K

April 12th, 2012 at 9:28 amJon Schwartz Says:

Good thing is just my dev server… This downtime is getting a little ridiculous!

Last straw. DH Support has been great, price is unbeatable, communication has been top notch, performance of my servers (when functional) has been awesome, but the service has been too unreliable for me to continue using DreamHost. As much as it is going to pain me to move all of my domains and their associated data, I go no longer stay with a host that has caused me so much loss of revenue. =/

This is appalling. There’s no point in doing the tired ‘I’m moving my business elsewhere’ routine but I don’t think we’ll have much choice soon. The outages are now so frequent and the complaints from our clients are becoming such a regular thing, that we may have to take on the massive task of migrating elsewhere.

PLEASE sort it out. Dreamhost used to be the place to go for reliable hosting.

Oh good, as soon as I go to launch a client site, their hosting is down. How embarrassing.

you gotta be kidding me – all my sites down??? happy dreamhost customer??? can you not have a plan b shit like that?? un bloody believable

WHY DO YOU HAVE SUCH AWFUL SERVICE??? IS THIS WHAT WE SHOULD ALL ANTICIPATE HAPPENING EVERY WEEK?!

Is this DDOS again ?

Down again!!?

This is getting ridiculous…. AGAIN give us our money back!!!

This is getting ridiculous

R-E-D-U-N-D-A-N-C-Y.

Well… shit happens…

does CloudFlare really keep uptime when data centers are having issues? ….anyone have experience with this?

WOW CMON GUYS!!!!!!!!!! I feel like my site is down eery day what the hell is this?

WTF? Again?
This is unacceptable!

Grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I’ll bet $1,000 this status site is hosted by godaddy – they cant trust their own uptime

balls.

my site is entered into a contest with a $2k prize.
bet my entry is looking pretty good right now

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?

This is ridiculous! Looks like I’ll be shopping soon for a better alternative. I’ve experienced more downtime over the past month than I have over the past two years!

This is unbelievable. Again!!! Two e-commerce websites down in the peak hours!!!

You guys suck, big time! *Moving my shit away*.

This is not the kind of uptime that a Level 1 Trauma center should expect. Our hosting bill is not small. Get us out of the Irvine center.

2nd time in 3 days, it doesn’t sound good!

Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost FailIncompetent! Incompetent! DreamHost Fail Incompetent! Incompetent! DreamHost Fail

That’s 2 strikes….

You’ve got one more!

Unacceptable.

This is completely, utterly unacceptable. I’m paying Dreamhost for a VPS, and this is what I get? What an awesome deal.

this is BULLSHIT you’d think that after Monday (and last week) they would fix the problem permanenly

Phone is off the hook with clients calling in complaining that our site is down. Do we need to look for a more reliable service?

Can anyone say bluehost????? Really getting tired of this crap!

This is unbelievable. MIDDLE of the day. Website just launched 10 minutes before this happens. I guess the old saying of “you get what you pay for” really rings true here.

this is unreal… for the amount of money we are paying for hosting, this has got to stop…or you’ll be losing a customer.

you need to have much better redundancy… unacceptable.

Again? 3 times in a row!!!

Seriously, Dreamhost is not reliable anymore.

Seeking for another hosting and VPS provider is a high priority task to do today

per: YES.

That’s what I am doing right now. 2 high-visibility failures in the same week, this is unacceptable.

any other host is not looking so bad right now

This is my fourth month with Dreamhost and my site has been down 3 times (that I’m aware of), I never had this problem Siteground in the year and a half that I was with them. Disappointed…

Actually Seth this is the third strike – it happened once last month also. Twice this month. I am losing thousands of dollars in sales. Please someone shoot me in the head.

You get what you pay for with DreamHost, right guys? This is the last straw, we’re moving to HostGator.

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?

anddddd the phone calls started.

Meh, let’s all just leave. What the fuck?!

And of course, let others know..

This is a little Last Straw-ish Guys!!!!! I was really riding with y’all too :-/

Here’s the funniest thing you’ll read today. I attended a WordCamp a few months ago and a Dreamhost employee was one of the speakers. His subject? Server Security! HaHaHaHaHaHa… The irony was almost too much to take.

Thanks ! we lost money again and again. I’m in France, and this is the time with lot of customers !!!!!!!!!!!!!

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

bluehost.
rackspace.

Once i could get, twice (well s**t happens)….. but three times? This just shows you didn’t fix the problem at all. Only patchy solutions result in this sort of inconsistency. Please fix it soon, I’ve enjoyed being with you guys for the past 5years…. but issues are starting to become a daily occurrence, not a monthly. If you cant wade through the storm of internet hacks and attacks should you really be offering web hosting? An Apology would be nice, but preferably some better fixes and redundancies would be better.

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING
LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING
LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING
LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING
LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING
LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

LET’S START HELPING EACH OTHER OUT AND LIST ALTERNATIVE SOURCES OF HOSTING

Now what is wrong… I need to write something for my blogs, and I mean right now. But all of a sudden, one of my VPS is down.

I’m missing some opportunity because of this. Sad.

How come this keeps happening a few times a week I almost want to find a new host. I am frustrated and so are the people trying to access my site :(

“The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!” Liars! Liars! Liars!Liars! Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!Liars! Liars! Liars!Liars!

“The DreamHost 100% Uptime Guarantee”…….

“The DreamHost 1% Uptime Guarantee”

Wow — the day I get linked by Gawker and other big sites and you people FAIL ME. WTF????

5 DAYS IN A ROW NOW! COME ON! I AM NORMALLY A PATIENT MAN BUT WTF!?!??!?!?? YOUR CUSTOMERS (MYSELF INCLUDED) ARE LOOSING SO MUCH MONEY OVER THIS BULLSHIT!

Anyone have any experience with Media Temple?

ive been with dreamhost since 06 and they get worse every month, i used to swear by them and now im becoming embarrassed that i ever recommeneded them, this is a serious problem were loosing hundred of dollars right now, im sure other are loosing more. WTF. There is no excuse for this, you cant claim to be a good hosting company and offer this kind of low rate service. I’m getting a refund and probably moving to media temple unless they can give me a good reason to stay.

last straw dreamers.

Come on guys!, fix this.

This is the 4th or 5th time since we launched 6 months ago. i appreciated the free hosting to non profits but i now think this was a bad move on going with dreamhost.

Lets get a STATUS UPDATE PLEASE!

Any Updates with what’s going on?

Please stop this!!!! I´m getting sick of issues!!!

关键时刻掉链子。。。

UNACCEPTABLE! What the hell happened to you, dreamhost? You used to be so good, now I can’t recommend you to anyone. Thanks for the day full of angry client calls!

my website is down, again… I made a mistake using dreamhost…

This is really killing my business. Please advise of the status. Why isn’t there a backup server available?

I agree, it would be nice to get more regular updates since we aren’t really at the end of the line. It would be nice to let our clients know what is going on.

visit my site!… hahaha just try it! I dare yah!

April 12th, 2012 at 9:41 amRichard Robins Says:

Dreamhost Y U NO Up?

joking apart twice in a week? this is worse than hostgator!

Figure out a refund / credit and I might not move the site. This is pathetic.

It’s been 20 mins since you posted this (and my site went down about 15 mins prior to that) – still no update?

hey guys my site continues down!!! what happen?

again???? this is just plain incompetence!

How is that I am using cloudflare and my sites are still down???

I’m going to give http://www.peer1.com a chance. They guarantee 100% network uptime to your server and 1 hour hardware replacement.

Really, get your act together or just fold up. For Christ’s sake.

This is really ridiculous-I want a credit for my hosting. The downtime is getting ridiculous-I’m running a magazine on your servers and if its down IM DOWN!

We have sent out a lot of sponsorship requests and those potential sponsors are now receiving the URL on the site to go to… but obviously that’s not going to be there, do you have an estimate? Is there a way to be notified via SMS or something, so I can… email those potential sponsors again with a mea culpa / estimated time? Thanks.

Confirmed… My site is down :-(

I want my money right back…

im a new customer to dreamhost and luckily for me im on a 2 week trial, it’s the second time in 3 days im considering canceling the payment that is due to go out after 2 weeks and switching to media temple

i’m dead !! Catastrophe! Mal chance! Barclock! poutouk! heee maleeeh!!!!!!!
During a meeting … and still not fixed!

“Comments posted here may not be viewed by DreamHost staff at all.” I read that as this being a place for users to complain loudly at a wall where nobody will ever hear it.

Unbelievable.

So far I see http://www.peeri1.com as another alternative.

It sucks to think I was giving Dreamhost a chance and like someone else, this is the second strike.

I agree with JIM Says. If you can’t get it together, just fold up. Don’t offer promises you can’t keep.

Very frustrating and more frequent than one would hope for.

I want to find the group or individuals responsible for the DDoS attacks against your Irvine Data Center. I’d love to give them a piece of my mind.

Why is there no backup dr site?

i cant’t believe this… this sucks!!!

haha: At this point we’ve verified that there is severe connectivity degradation in our Irvine datacenter.

I verified that 45 minutes ago!!!

“At this point we’ve verified that there is severe connectivity degradation in our Irvine datacenter.”

CALL NICHOLAS CAGE:

“YOU DON’T SAY?”

LOL

I agree with Deltrice. Let’s get Clifford Stoll on the case

What’s even worse than DreamHost is us, the suckers who are still using them. We are the enablers.

“This is getting ridiculous.” Agreed.

Do we have any legal rights to a refund?

Just intern host this shit!

UNACCEPTABLE. Great way to sink your company. Looking into alternatives now.

I don’t even get surprised anymore… can this really keep on happenning? what about cloudfare? wasn’t that suppose to prevent sites from going down even if the server did? …

Just launched a new website and advertising push. This is the worst possible thing that could happen!!!

“we’ve verified that there is severe connectivity degradation in our Irvine datacenter”

I think this is worthy of beating a dead horse (or dead network connections…)

In the imortal words of my highschool classmates

NO SHIT SHIRLOCK!

pfff, figures.. this ship is sinking fast

We’re moving away to Hostgator tomorrow. This is ridiculous and I’ll make sure to always bitch about you in every single hosting conversation for the rest of my life.
You just cost me about 3000 dollars.

YOU GET WHAT YOU PAY FOR JAJAJAJAJAJA, thats why I would rather pay top dollar for a server than 10 bucks:

*************************************************************************************************************
The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!
****************************************************************************************************************************

I sure hope y’all took advantage of that guarantee right there, I repeat “We begin assessing credit from the time you open a support ticket to report the problem.”

If you didnt oh well

I’m going to go flush my head in the toilet and hope that, if I am not drowned, that my websites will be up again. I am just thankful to Google that they have not make my search visibility suffer for this. However, I’m sure that Google Search is aware of DreamHost’s problems, and eventually their poor performance could become a negative ranking factor for us all. That’s something for us all to keep in mind.

How long will it take to solve the problem? Can anyone give some information?

I have moved my primary sites to http://vps.net their interface is nothing close to how awesome dreamhost is and the ease that you can add sites/domains cannot be matched by anyone in the market… the uptime issue means i’m moving another batch of my sites this week to VPS.net

Yayyyyyy! Back up!!! I love Dreamhost <3 <3 <3

Any recommendations on how to make a seamless hosting change? What hosing company should I switch to? I can’t keep doing this.

NIGHTMAREHOST.com is more suitable

We have been down for 7 hours!!!! Our website has been offline that long, and then it is no comfort that our SQL- or mail-server runs :(
When will this be fixed??

This is the third time for the Irvine datacenter in a week. How can you continue to do business with the people in this datacenter if they cannot keep a stable server environment for more than a day at a time. I mean, it just seems like you need to change datacenter locations, or get new people to manage the datacenter. Not that hard a concept to understand. If I don’t get my job done, I get canned.

who designed this data ceter? do they only have one link and firewall to take down? redundant routes from different sources? redundant firewalls? that’s standard tech for any large business

“The DreamHost 1.00% Uptime Guarantee.”

Top notch team determines there is a connectivity problem. Sounds like we are in good hands. I am sure within a few hours they will discover that some websites are not working as well.

Metro Goldwin Meyer Sites are in DH in IRVINE………
thinking……
3 times in a month
Still thinking……..

dedicated server lift and shift? or virtual host copy/port

Step 1: Unplug the router and the modem.
Step 2: Plug it back in.

That usually fixes it…

who designed this data ceter? do they only have one link and firewall to take down? redundant routes from different sources? redundant firewalls? that’s standard tech for any large business

Can’t believe it came back up so quickly this time

Sweeeeeeet, back up!

so how many hours was it down for this time?

09:56am PDT and still down for me…. 2nd time I’m aware of this month for me.

http://www.ring-rap.com is down. Again. Just switched to dedicated last week too. Thanks guys. :-\

still down for me

WTF we are not back up yet…….

Just got an email sayin they only got some back up…?

53% Shared
62% VPS
3% Dedicated

apparently?

ok thats it, i moved to iweb.com, fuckk youu!

Back up now….. lets see for how long….

Okay that was fast thank god.

Mine’s back.

just now came back up (09:58am PDT

Back up currently. Really wish my server was in a different datacenter…

ohhh… no… que susto… no puede ser… por favor solucionenlo pronto..

I just signed up with you guys last week. First impressions are lasting impressions… Hope we’re not off to a bad start.

Me 2!!

My site is down and I cant have my people work on it. Loosing time and money.

I will be expecting a refund

So like,

I was coding, but now I’m on break.

please fix soon.

Re: I love

Hold your tongue . I am losing money every second this shit goes down. My 6 sites are still down and I have a presentation to give in 4 minutes that is stuck on my FTP.

Any DDOS douchebags need to realize that this is serious shit and it hurts a lot of people. They need to go find jobs or go back to playing WOW.

YOUR SITE ALWAYS SEEMS LIVE DREAMHOST.

WHY NOT MINE?

I am going to have customers call you directly if this continues!

Be’Jesus!

ya estamos de vuelta.. ya puedo ver mi página muchas gracias…

WORKING…. thanks

Dedicated server looks to be back up here, thank god.

Quickly loosing money here!

I am taking my business elsewhere. This is ridiculous.

Next time pay the bills!

April 12th, 2012 at 10:04 amithinkitsnice Says:

So you lot are the new fasthosts then? Is that it?

And now it’s down AGAIN… 5 minutes uptime lol!!

So glad that I moved my site to a new host about a month ago because of the previous issues. Before, I had a MySQL and Web PS and was paying about $250 a month in total to DreamHost because it is a big site.

I was just checking here to see if I’d made a wise decision, and what do you know?! WOOT!!!

Total downtime on my new host since I left DreamHost? NONE.

…And I am paying less money! :)

We were back up…for 3 minutes. I guess the Duct tape fell off again.

finally returned

Spoke too soon, it’s down.

Back up,

Quickly! Everyone finish as much work as you can or try to pull in as many sales as possible! We have a short window of Real, Actual UPTIME! HOT DAMN! I estimate we have about 10 hours at the most… maybe worse lol.

This is absolutely maddening, STILL NOT BACK UP ….

Feels like a roller coaster! UP DOWN UP DOWN

I want off this ride.

Back Down :(

Well that was a short lived window.

So let’s start a new topic, which company are each of you switching to?

April 12th, 2012 at 10:07 amBilly Ray Cyrus Says:

THE HOST IS A LIE

ICDSoft for me.

I WAS UP. AND DOWN AGAIN IT GOES !!!!!!!!!!!

dammit… down again

Switching to the first one that offers VPS with unlimited HDD…

DOWN AGAIN!!!!

Maybe HostGator…but maybe any host with the word host in it is a bad sign.

same here, up for a few minutes, should be called Nightmarehost

We’re looking at WestHost’s Monsoon Cloud Servers .. anyone have any reviews?

UP AGAIN!!!!

Thanks for working on this, guys. Sorry to hear you’re having difficulty. Hopefully this will be resolved soon. — Mike

I like how the updates have times that are in the future, “Update Apr 12, 9:42pm PDT” – can’t even get that right. /clap

Come on Guys!

Have you ever considered taking up decorating?

Cos you are crap at hosting!

“Enterprise-Level Reliability

Our expert technical support representatives are available 24/7 via email, live chat, and you’ve even got the option of reaching a human being over the phone. Our 100% uptime guarantee means we take uptime seriously.”

As serious as Whitney Huston took her coke problem…

Could this fall under a class action? Just currious.

So it’s Not moving good…..So I’m Not considering it back up yet :-/

one more time up… and down again

AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!! AWFUL SERVICE!! DREAMHOST NEVER MORE!!

LA 2, Irvine 0 – this week

Yeah,

My site was up for 90 seconds!!!!!!!!!!!!!!!!

down again !!!!!!!!!!

New game: Who can get the most work done in the 2 minute intervals of up-time.

Seriously, I have never had this issue with a sever before shutting down for so long, in prime time back to back… is this an ongoing issue?

Switching to PHPFog + Amazon S3 + Amazon EC2 here. This is unacceptable.

hey guys. seriously i don’t even have enough time to keep responding to my own clients that are furious with their websites going down. It makes ME look bad for choosing you. would really appreciate a resolution to this, or a refund to my account.

Nunca vi um serviço tem bosta como este!

down again

Where the hell is my SHELL ACCESS!!!!!!!!!

ta pior que a Locaweb, rsrsrs

Not out of the woods yet. Site is still continually timing out.

Take advantage of any up-time you get to upload a redirect page to your new host.

Ok, so, I was nice last time (what was that, tuesday?) but 2 times in a week for more than 30min? Now it’s impacting my svn hosting. While i’m not losing income from a website being down, my subversion hosting is ruining my integration schedule right now. Please update us and fix it asap…

Your service is so wonderful!

Does dream host even read this ?

Despite my fowl mouthed fellow hosters comments, I am thankful to Dreamhost for the service and realize that things happen. Thanks for getting things back online quickly.

While my site is down i’m using the time to move to Amazon EC2, thanks Dreamhost! Never looking back!

Patrick, I’ve lost around $7k over the last 3 months alone due to DH downtime. They can go fuck themselves.

E ai brasileiros.. Alguém ai trabalha com o Magento? (SOMEBODY WORKS WITH MAGENTO?)

I have been up and down now so many times.

Seriously? I’ve only been a part of this service for a few months and it’s been hell the entire time. The first month I had joined was the worst month you ever had (stated by you). This better be fixed soon I’m spending money to be hosted not spending money to have my site dead.

That’s it… game over… Dreamhost done… Moving my service today…

I am paying employees that are just sitting around because they can not work on sites with DEADLINES.

I have paying customers SCREAMING at us because their STORES are CLOSED.

We have presentations that we can not give to prospective clinets. WE ARE LOSING MONEY TOO!!!!

You guys really suck. You know you are the only network incapable of keeping the network up?

losing money every second!!!!

yawn! lets go guys….i think everyone here is thinking we pay you enough to be on the ball and up and running….not to mention a back up system?????

Why can every other web host keep their sites online and you can’t?

I was freaking out thinking I had done something to our forum. Good (well, not really) to see it is web hosting problems. It does seem to happen too much over the last few days. I’m glad we are just a forum…not an e-store!

Good luck DreanHost…and hopefully this will get you to make the servers more secure.

This is the last straw for me. I’m moving to Blue Host. Anyone know how to move a website? Are there any tutorials available online? I don’t want to pay someone to do it and my site is only medium sized…

up for these next few seconds, treasure them :D

wow, just wow. I’ve never heard of a hosting company with as may problems as DH. I’ve got an email campaign driving traffic to my site as we speak, that means I AM LOSING MONEY
thanks DH

Thank god I just bought a cloud on Rackspace, i recommend everyone jump off the sinking ship that is dreamhost until its too late. Ultimately its US who our clients look to and blame for their websites being down. And when their website takes a shit at least once a month, they will not stay your client for long. Do yourselfs and your businesses a favor and find hosting elsewhere, I have been a client for a long time so I am qualified to say “it sucks, and for me, its always sucked” Goodbye Dreamhost, i cant wait for the new nameservers to populate.

Waiting for service to resume is like watching your grandad trying to get a hard-on for his annual birthday treat

“Come on old chap, you can do it…”

GAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? MY SITE IS DOWN!!!!!!

So Dreamhost got its name because if your site is running it must be a dream?

lol is true!! this sucks!!!!!!!! but is a dream…. is a dream…

DRAM TO BE UP HOST = DREAMHOST

dream to be down…. all time… i can’t believe this…. !!!

Last straw for me too. Just signed up with mediatemple and have started moving sites. The last year has been horrific with dreamhost. Not only do we lose sales when the sties are down, but we have seen a loss in search engine rankings as well. Good. Bye. Dreamhost.

Dreadhost? lol

still not fixed went down 4 times again it works for 2 min tops then stops wow…this is really a dream service!!! why do i pay so much money again…plus no cutover service wow i feel like going to network solutions and pay even more just to talk to someone!!!!!!

We’ve moved one of our business critical Rails apps over to Heroku where we’re seeing great uptime so far (been there ~a month). Heads up if anyone is looking for backup servers or apps cloud hosting, etc.

good bye too.. i have to find quick solution..

Lucky our marketing push doesn’t start until the end of the month… hopefully everything will be stable by then.

Irvine DataCenter :: Irvine DataCenter :: Irvine DataCenter :: Irvine DataCenter :: Irvine DataCenter :: Irvine DataCenter ::

The shittest hosting alive, these people are on that old man shit…..falling apart cause you just suck. I hope all potential customers read this and see that our hate is after months of crappy service and yet this company does not for their customers. I eel like blowing up your headquarters like ending of Office Space….

okay….we’re done here. This is complete BS…why am i paying for a VPS when it is constantly down. Time to migrate.! Bye, by dreamhost

????????????????????????????????????????????????????????????????????

April 12th, 2012 at 10:47 amNigel J. Terry Says:

Again! This is ridiculous. Serious companies have redundant networks and servers. What other sites do people on here have experience with? Bluehost, Hostgator,MediaTemple?? Just don’t say SlowDaddy, that’s where I was before

Hahahahaha:

“Lucky our marketing push doesn’t start until the end of the month… hopefully everything will be stable by then”

To be so innocent :-)
Leave while you still can

still dead!! jizzzz! fix this sh*t quickly!

April 12th, 2012 at 10:48 amMylan Connolly Says:

We run a fair amount of stuff through Dreamhost, and while it’s not super mission-critical stuff (which is why it’s on a VPS in the first place), all the down time we’re getting lately is really hindering productivity.

We have been testing the waters with a competitor, and this last outage has pushed us to a complete transfer, just so you know.

Over the past couple of months there has been a SERIOUS degradation of service. Might be small potatoes to DH but I have two accounts paying roughly 500 USD a year. With this event, I am working on moving them.

This is just pathetic, we seem to get all manner of problems week on week. Our developers are moving and taking all their clients away from Dreamhost. We moved on their suggestion because you were so “reliable” so far nothing but trouble week after week-awful, awful company

500 USD YEAR? I PAY 1.188 USD / YEAR … My website have 400.000 visits per MONTH, I CANT HAVE MY WEBSITE DOWN ANYWAY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Rackspace cloud! Who wants my reseller code?! :)

Well this is it. Second time this month now. Im now leaving for a better host.

Take care!

And new users run and dont look back!

Can’t figure if you should be called Dreamshit or Shithost

Grrrrrrrrrrrrrrrrrrrr!!!!!!!!!!!!!!!!!!!!

PLEASE FIND A BETTER SERVER THEN DREAMHOST AND TELL ME HERE!!!!!!!!!!!!!!!!!!!!!!!

ugh, on again off again. 5th time this week my domain is down, 3th time today. I’d much rather have it be down slightly longer and have it be fixed, than it going up and down every hour.

This is terrible. I am moving my sites. I have been with DH for 3 years and nothing like this has happened. Any recommendations for new hosting companies?

http://www.bestvpsproviders.com

Thought you guys might want to take a look.

PLEASE FIND A BETTER SERVER THEN DREAMHOST AND TELL ME HERE!!!!!!!!!!!!!!!!!!

This company is showing signs of a failing business-it seems it can’t afford to keep a level of service up. What will happen to all those stuck when they go under-the previous comments of sinking ships may have something! My advice, leave before you lose even more business.

COME ONNNNNNNNNN.

PLEASE FIND A BETTER SERVER THEN DREAMHOST AND TELL ME HERE!!!

Just signed up yesterday and got my landing page in order. Come back a few hours later and it’s down. Not a good start.

How do I get a credit for all this downtime?

I’m so tired of you guys costing me money. You used to be so reliable and lately it’s problem after problem.

Speak English Oscar, WHEN IS THE NETWORK GOING TO BE FIXED???

The header of the page says What’s up

Should read What’s down.

Ditto everything above. I signed up for a dedicated server thinking I wouldn’t have to worry about the downtime issue any more. hahaha

Dreamhost, please move my sites to wherever your site is hosted since yours seems to be up 100% of the time.

Oh, and one more thing, wtf with the Live Chat? You expect people to pay for support when every other hosting company offers it for free. Your customer service has really gone to shit. You realize cutting off communication and ignoring the problem isn’t going to help you retain customers.

Its down again? LOL

DOWN AGAIN.

PatrickD you’ll never get a credit, it looks like they’re in trouble

What hit the servers?

Ok…. More updates with clear explanations would go a long way to retain clients at this point

This is a nightmare. I thought FatCow was bad, but Dreamhost is constantly down! I just pointed someone to my website and guess what, it’s not up!!! You should provide a monthly credit back to everyone who hosts with you, as it’s amazing how many days you have service issues!

Nah, we’re not innocent. Just hopeful. We’ll monitor and jump if we need to. PLENTY of alternatives (are you listening DreamHost?)

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Maybe your test team uses the same routers and have flooded your network… or maybe you are not testing enough.

Come on… nail it down!!.

Cheers

funny thing is that i switched from jumpline to dreamhost just recently because i knew dreamhost has more options available (which i use for other sites/customers i have w drmhst). Guess what: i am switching this customer right back to jumpline and expect my money back from dreamhost

Do we get reimbursed for the down time this is fucking ridiculous

This is killing me. May not have a job tomorrow.

This is incredible, 3 times a week, what is the plan to resolve once and for all the damage?

I just want to be moved to whatever server is hosting this status site – alls good here

We used to have our sites on DreamHost. Moved a bunch of them away.

WiredTree, LiquidWeb, VPS.net, … we never have these issues on them

April 12th, 2012 at 10:59 ambuythiscomputer Says:

***************************************************************************
***************************************************************************
***************************************************************************

DREAMHOST YOU NEED TO INFORM US A LOT BETTER THAN WHAT YOU DO ABOUT THE REASONS OF ALL THESE OUTAGES !

At least give us a chance to understand why is your service so bad lately ?

Is it a problem with your staff ? With your hardware ? YOU SHOULD EXPLAIN AND BEG FOR PARDON !

***************************************************************************
***************************************************************************
***************************************************************************

I can`t. Anyone will pay me for that?
I will lost my client`s.

Amazing panel – SH*TTY network!!!

BIG CRAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP

back…let’s see for how long

we’re leaving, these people are incompetent

I am looking at switching providers!! A whole day of office staff waiting for our off site database to come is RIDICULOUS!

DreamHost is terrible guys. Please do yourself a favor and move. They actually forced us to move. They said they are done with us when we asked for a refund for a week of downtime. haha.

Just purchased an account over at BlueHost. I have to say I feel better already. Dreamhost has been nothing but headaches from the beginning – even aside from the downtime. It was nice to actually CALL BLUE HOST and talk to a LIVE PERSON. Wow what a difference. And they are cheaper…

4 calls so far from clients informing me they cant see site, that = lost business, lost income, lost money. GET MY SITE UP!!!!!!!!!!!!! NOW!!!!!!!!!!

SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!
SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!

SOMEBODY GIVE ME NAME OF DEDICATED SERVER?!?!?!?!?!?!!?!!!!

The grass is always greener…

Server Sadalsuud is On!

5th time being up and then down today,

everyday this week my clients have gone through downtime, one of which (my biggest) had downtime everyday.

I am pulling each of them from Dreamhost, and will never recommend this company again.

I urge all of you to vote with your reviews to ensure others won’t have to experience this debauchery again

Is the ‘sulafat’ server part of the Irvine Data Center? Because my ftp and webserver is also down but I’m not sure if it’s because of this.

I am thankful my website isn’t business critical, people are actually losing money from this and that’s simply not good enough.

You’re causing me to let my regulars down by not providing the service they expect, and you’re letting yourself down by seriously damaging your reputation.

>”high CPU on our routers as well as that of our upstream providers so we’re endeavoring to move some routes manually on our end”
TRANSLATION: You’re running around the comms room with ethernet cables in your jimmies, that’s hardly a sound redundancy plan.

FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING! FIX THE FUCKING THING!

K.. I’m done… Migrating.. Bye bye Dreamhost

moving to mediatemple.net this is horrible.

Site is back up – 2 hours later. Nice guys finish last. Thats me!

correction.. site(s) back up. Buy pins make me feel better.

I am at the point where I cannot in good faith ever recommend Dreamhost to a client anymore, even for small sites.

We shouldn’t have to put in a support ticket in order to be compensated for downtime – you should be giving that to us automatically.

I find it amusing that people say they are going to quite but they still stay with DreamHost. These guys know a lot of people who say they will move won’t. That’s the problem. These guys have a horrible service. Have had it for years. Certainly not to be trusted for business. We have tried all their plans. VPS, Dedicated. Their security breach from a couple of months ago should have pushed a lot of people to move but didn’t. It’s sad. These guys won’t innovate unless people really move

No Rob some of us are still down!

And up and down and up and down and clients calling and “sorry no idea when it will stop”

okay nevermind -Sites were up and now they are down again. What a productive day this has been. Maybe I could’ve done more for my family by waiting in the food stamps line.

Cyrus-our developers are moving ALL their clients and sites, a lot!

Well if not to be trusted this is my first host and not even a year who is the best to go with?

I hate the fact that my VPS always goes down yet other websites I help out with on DreamHost shared accounts are up and running. They actually seem to have more uptime than my expensive VPS. WTF?!?

Clients are in full panic mode – been down for far too long now. Nervously telling them that Dreamhost will have the problem fixed soon. Fingers crossed!

Dude, I love those Pins Fast designs

@aj. Good for you guys! I think we all should. I have moved a ton of sites to VPS.net and will be moving everything else, even the non important sites away. Until customers use the free market principles to teach companies with bad service a lesson, they don’t change. I like DreamHost. But a dedicated server being down two or 3 times a week is just not cutting it.

Is DH paying us for the damages these downtimes create?!?!

Thanks Aaron! Are you moving to a new server too? I am calling Rack Space right now. Maybe I can charge it. LOL.

And…. it’s down AGAIN!!!!!!!!!!!!!!!!!!

Agreed Cyrus!

Stop Whining you big babies.

Does everyone knows hosting company with offer for VPS with unlimited HDD and Bandwidth?

Guys, i know you’re getting a crapton of negative feedback… my livelihood isn’t depending on this site like others are dealing with, but i put all of my heart, soul and free time into it and right now i can’t talk about it or send people to the awesome content i’ve spent what little free time i have creating because it’s still down after 2 hours. I don’t understand what a “flapper” is, though i’m pretty computer/server savvy. If you could just say which servers are effected and how long it typically takes to fix these issues that would be helpful. I appreciate the updated statuses though.

Sorry if this has been asked already. Anyone have a good alternative for dedicated server hosting?

LOL I am not whining just pissed and if I pay for something I expect to get it! :P

April 12th, 2012 at 11:33 amShawn Gossman Says:

Personally, I have not seen one host with 100% uptime. However, this data center issues is real and needs to be addresses. I pay over $100 a month and have been with DreamHost nearly 10 years. I hope to stay another 10 years but wow, move me to a different data center, haha.

Don’t worry about it. We’ve already moved our sites off of DreamHost. Buh bye.

we are all too lazy to leave them and they know that and its why they don’t give a shit about their service

This really stinks, it has been 3 hours and I have not gotten anything done on my site because it is down! When you only have a few hours a day to get things updated, 3 hours kills your day!

2+ hours now….

TICK. TICK. TICK.

Well at least they guarantee 100% uptime….

Make sure they pay you back what they said they’d pay you back. Each and every one of us needs to make sure we at least get that money back.

You’ve just lost three customers. I’m migrating everyone away once we can ACCESS OUR CONTENT.

What does it take to get my server off this data center?

I just transferred my WordPress blog and based out of L.A. Is it safe to assume this is my data center? My site’s been down for 2 hours too

2-3 hours? SAY 9 HOURS!!!!!

I’ve had great experience with mediatemple, they do go down but you can always get a person and an exact explanation of whats going on, similar pricing, they run plesk, which is more annoying then dreamhost’s panel, but still it works. Thier vps service is also fantastic, i had a server crash for 20 minutes once and they talked me through my panic and gave me a refund, dreamhost used to do that too, but they dont anymore, support is almost non existant, WHAT HAPPENED? Did you fire half your staff? get some new incompitant managers??? i dont understand how such a good thing went so very bad.

I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND. I WANT A REFUND.

still down..almost 3 hours.

Thinking about Amazon EC2

wow, can’t believe this is happening..AGAIN

“As we see these “flaps” we will manually reset the connections and services will be restored”

Great, how about a manual reset now. Restored service sounds good.

I would like a solution – how can this be avoided in the future? Why don’t you have a backup server/data center?

My site is down. Guessing this is the cause?

AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???

ALL network administrators and interns are dumb, stupid people just have it!

I guess our VPS is on a “flap” as we are still down. I’ve been a supporter for years but professionally can’t take the heat that Dreamhost brings anymore…

AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???AGAIN???

Funny how DH’s Terms of Service contradict the promise they make elsewhere (where it is actually visible) on their website. While they promise to “credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption”, their TOS states that “DreamHost is under no obligation to compensate Customer for downtime, whether the downtime be caused by Customer, DreamHost Web Hosting, or DreamHost’s upstream providers”…

Isn’t a critical component in a toilet called a flap? or is that a flapper?

Dedicated server still down.

Well, I don’t really know what you mean by “flaps” but I know my site is down still and it is frustrating to know you don’t seem to be able to stabilize the problem.

ZZZZZZZZZZZZZZZZZZzzzzzzzzzzzzzzz

Well Ed that seems like total bullshit especially since they specifically tell us

That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

Not to worry about fine print because it doesnt exist. So Sorry to dreamhost but I’m going to ride their asses until I’m fully refunded or at the very least get the money they promised me for shit like this. And if not I’m going to use their hosting to attempt to host a site telling any potential customers to stay the hell away from this place.

please help…. still down… crucial applications …

Our site is still “flapped” to the off position right now. Any chance you can get the manual flapper person to flip it back on right quick?

This affect the NAOS (shared host) too???? All my 7 domains in NAOS are DOWN…. PLEASE HELPE-ME!

Echo echo echo echo

zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

thats enought!!!!!!!!!!!!!!!!!! fix the damn problems’!!!!!!!!!

Flap Flap Flap Flaperoo

Seriously? so everytime the “plug” falls out some poor shmuck is going to pick it up and plug it back in? That’s the solution here?

Watching the dollars float out into space instead of into my business.
: (

Posting on their facebook page would be terrible publicity

Down again. VERY BAD TIMING. COSTLY!

Your update references to “periodic” connectivity issues would imply that this is less than a full-blown outage… but my websites have not been “periodically” unavailable. They are down… PERIOD!

In computer networking and telecommunications, route flapping occurs when a router alternately advertises a destination network via one route then another (or as unavailable, and then available again) in quick sequence. A closely related term is interface flapping where an interface on a router has a hardware failure that will cause the router to announce it alternately as “up” and “down”.

Route flapping is caused by pathological conditions (hardware errors, software errors, configuration errors, intermittent errors in communications links, unreliable connections, etc.) within the network which cause certain reachability information to be repeatedly advertised and withdrawn. In a network where a Link-state routing protocol is run, route flapping will force frequent recalculation of the topology by all participating routers. In networks with Distance vector routing protocols flapping routes can trigger routing updates with every state change. In both cases, they prevent the network from converging.

Route flapping can be contained to a smaller area of the network if route aggregation is used. As an aggregate route will not be withdrawn as long as at least one of the aggregated subnets is still valid, a flapping route that is part of an aggregate will not disturb the routers that receive this aggregate.

We host 182 client sites at Godaddy

Total downtime for all 182 Godaddy sites in last 4 months = 6 minutes

We host one site here and the downtime is 4 HOURS (so far)!!!!!!!

Now calling these guys NIGHTMARE-HOST…!

Don’t suppose I could just offer help, I’m a CCNA, and senior network admin.

lot @ people asking for refund. Do you honestly think Google will care about how much refund you get from DreamHost if your site is down all the time? Or your lost visitors will come back if you get $100 back from DreamHost? That’s why these guys are awful. Saying you will refund people for 24 hour downtime won’t bring back lost sales, customers, credibility, rankings… I agree with @John. Moving takes a lot of effort. I was lazy for too long

“Leaders in web hosting, virtual private servers, dedicated hosting, shared hosting and cloud web hosting.” … please.. attend us

How do I get a refund when I cancel? I’m going to have to move to a new provider and it is going to cost me.

I’m moving our e-commerce site to inMotion.com – I’m about 90% finished, I still need to do some additional testing and then coordinate the transfer. It’s a shame really, Dreamhost has been around a long time, and I was there at the start – it used to be special.

“Hi, Kevin! (Member since July 2000)”

Second time !!!!

Any suggests of a good replace at same price for dreamhost? [Dedicated Server]

How is it that you can’t keep up Dreamhost? — Mediatemple can… Hostgator can… even Hostmonster, JustHost… just to name a few. I’ve got sites in all of those and only YOURS ever go down.

Telling us that you’re working on it is great… but you realize that you’re *always* working on fixing these issues… time to replace your old equipment and take preventative action. We’re all sick of this crap!

j I’ve already been on their FB posting away. As everyone else should.

Cyrus I’m asking for a refund so I can move everything I have to a new host and not worry about the lost money here. Full refund here = me going to amazon ec2 and hosting my sites at a place that will actually host the site.

April 12th, 2012 at 12:06 pmNigel J. Terry Says:

I was up for a few minutes, now down again. I guess I got flapped!

Don’t be lazy, MOVE, as soon as possible. These “flappers” are ruining our businesses! Post on their fb page too to try to get them to pull their fingers out of whatever hole they are stuck in!

Please, solve this problem. I can’t have my site down! it is my business!!!

‘Unable to connect database problem’ stills continues.
Our site is down for a long time. Is this a good company policy ?
We lost a lot.Pls help us urgently..

Continued patience…. I think everyones patience is long gone Ralph.

If any of the customers complaining here need some help. Contact me, I’ve been moving my sites and other client sites off of DH and I’d be happy to help you as well. — notagrouch.com/contact

Hey Oscar in the process of moving did you manage to get any refunds out of these people? Any info on that. The moving isn’t much of a concern for me as much as the money I spent on apparently nothing.

: We continue to see disconnects but we are making ground with some prefixes now working correctly. We are making progress with resets. We thank you for your continued patience as we continue to work through this incident. – Ralph

How about some money back ralph

To be fair, would you want to stop playing table tennis in that fancy kitchen with the great views to go much around with some route flapping issue?

Didn’t you know we’re not paying for service we’re paying for ping pong tables. We just don’t get to use them.

Moving my 10 servers to Amazon asap. I’ve also heard good feedback about Rackspace.

I thought purchasing the DH dedicated server and VPS’s would prevent these types of problems, or at least they would rarely occur. But outage is a regular occurrence at DH.

My web site is off all the days!!!!
please….

What happened to Oscar? We want Oscar back, Ralph is too new to know anything! Go away Ralph, get Oscar back

@Steve. Sorry. Didn’t mean to sound condescending.

If you need to move, try Liquidweb.com, Wiredtree.com or VPS.net. These are not affiliate links and I don’t work for them.

I wouldn’t trust web host review sites. They use affiliate links but more importantly push dubious hosts if they pay more. The above businesses offer superb customer service. Read forums before signing up with any host. I wish I had done this with DreamHost :)

my wife have birthday on 13 apr. I designed site for her as a gift…and when time to launch…its DOWN!!

up…fix it sonner.

thanks,

all sites are down, really really bad timing by you guys considering this is our best sign-up period.

Issued my support request as per the 100% up time guarantee now i sit wait, its concerning seeing so many outages all at once when i never had one with my old host.

Ill give you a chance DH but please get her back up

I don’t have any patience ! I am losing money..

Now it’s okay! Thank You Dreamhost for down my sites for 2 hours.

本来以为是我自己的问题,原来大家都down掉了……还好我的只是博客站不要紧。。。

But anyway I paid $9 a year! I need it fixed ASAP!!!

@Cyrus no worries we’re all frustrated here. And yea I actually got this site recommending to me by someone I work with. Apparently he’s brain dead OR works for dreamhost on the side. Not sure which. I guess I’ve learned to get to know someone before taking a recommendation from them regarding hosting.

@BF don’t speak too soon it’ll be down again shortly if ours is anything to go by

Let’s try to be a bit more genuine with the next update:

Update Apr 12, 12:23pm PDT: We know you are pissed and we know this sucks. We realize your are totally fed up and out of patience but we really are trying to fix it. Please don’t leave us, we don’t want to give up our commercial grade espresso machine in our gourmet office kitchen. – Ralphscar

Hahaha I love the website link Wall Yeller.

Just FYI, everyone, there seems to be a major regional outage near Dreamhost’s datacenter. From what I can tell, the outage affects a large number of sites that are physically hosted near Irvine, Ca. It’s not just Dreamhost, it appears to be a major regional internet backbone outage.

Matt,

Thanks. Would have been nice if DreamHost told us that – rather than telling us “At this point we’ve verified that there is severe connectivity degradation in our Irvine datacenter”. Thre isn’t a person here that doesnt know that….

Lol, I find it funny that they have this comment form available, just for angry people to vent.

I’d hate to be the sysadmin there. Or, maybe he/she isn’t stressed at all. Just kickin it eating some chips and guac.

http://www.internetpulse.net <– no internet based outages.

Let the countdown to downtime begin.

agaaaaaaaaaaaaaaaaaaaaaaaaaaaaiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiinnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

At flapping last, yes let the countdown begin imbeciles!

Somehow dreamhoststatus.com is up ALL THE TIME. God forbid your customers’ sites have that stability. I guess it’s back to godaddy.com for my sites.

Yeesh don’t do that to yourself. Godaddy sucks too. Check out something new I’ve heard good things about amazon ec2 but I haven’t used it myself.

My sites are back up. Eh, outages happen. Good work DH.

My site are back up too, Thanks !

10 HOURS OFF!!!!! :(

Don’t waste your time whining people … change hosting provider!!! …seriously. Then maybe you’ll realize how good dreamhost is and you may come back. There are … and there will always be technical issues … here, there and everywhere but responsiveness is what it matters and those dreamhost dudes are good at that (at least they are sufficiently honest and brave to inform us and cope with your baby rants). BTW, I don’t work for dreamhost nor receive any money … I am customer that pays as much as you do but I have worked in customer support since many of you were still wearing diapers.

My website is working again thanks ..

The peering connections have stabilized, all routes restored, we will continue to work to ensure stability and will manually reset connections if there are any further disconnects. We will continue to update this status post until further notice and we are 100% assured of stability.

Uh, maybe this should have been done when you guys first started….

Anyone else still have no connectivity?

JFK, Great advice. Most of these are not ‘baby rants’ however, a website going offline is an extremely serious issue. And a server hosting thousands of websites going offline for 4-6,8 hours at a time regularly is UN HEARD OF. Serious real world money is lost every minute in some cases so until you work your way out of customer service and into the c suite, or venture out on your own, leave the lecturing podium.

SECOND TIME THIS WEEK !!!

down again guys, starting to get frustrated now

and were back up, yo-yoings better then stable but still frustrating to our users

better then unstable/down i should say >.<

Still having piss poor performance on my domains that are cause timeouts in my app

DH service is intermittent, sporadic, unreliable and atrocious. Shame on your fucking ridiculous asses.

Gettibg tired of apologizing to my customers.

Please can anybody direct me towards a better hosting company? Also to a site where the Top Hosting companies are reviewed along with a list of their outages? I have lost an entire day. Where is the compensation?

@Jerry: Thank you for your constructive comments (one of the few) but “…most of these are not ‘baby rants’..” (most are, sorry) “…a website going offline is an extremely serious issue” (I never said the contrary, I have dozens of them) “…And a server hosting thousands of websites going offline for 4-6,8 hours at a time regularly is UN HEARD OF” (did you hear it was a regional issue?). “…Serious real world money is lost every minute” (mine is not “monopoly” money) “…until you work your way out of customer service and into the c suite, or venture out on your own” (if you own a business as I do then you should know that you can never work your way out of customer service) “…leave the lecturing podium” (actually, I agree on this one).

@JFK I own a few companies myself, but everyone knows that nothing has 100% uptime, even with cloud hosting it has issues. I appreciate the customer service I get with Dreamhost, so that is the main reason I stay with them. You won’t find that with Rackspace or GoDaddy to name a few.

This would be a great time to do advertising shouts…I notice that some people are putting sites up that they are moving host to “weburl” and the link is a parked page or some cheap design lol!!!

The humor 400+ comments brings…now back to work!

@Travis. You won’t get that with Rackspace but you will with WiredTree and LiquidWeb. Of all the hosts I have been on, those two have never let me down in customer service. Also, if you need support for your server, I recommend https://cliffsupport.com/

I don’t work for any of these guys by the way. I just use them ;)

It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!
It is not clear if this Oscar (Dreamhost), is a complete idiot, or a big asshole!

Site is not loading AGAIN at 3:00 pm

Sites still going down and not working

Site is down aff

Reply from 208.113.226.219: bytes=32 time=83ms TTL=50
Reply from 208.113.226.219: bytes=32 time=141ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=85ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=85ms TTL=50
Reply from 208.113.226.219: bytes=32 time=142ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=88ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=84ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=84ms TTL=50
Reply from 208.113.226.219: bytes=32 time=140ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=84ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=107ms TTL=50
Reply from 208.113.226.219: bytes=32 time=163ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=88ms TTL=50
Reply from 208.113.226.219: bytes=32 time=148ms TTL=50
Reply from 208.113.226.219: bytes=32 time=150ms TTL=50
Request timed out.
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=121ms TTL=50
Reply from 208.113.226.219: bytes=32 time=156ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=142ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=87ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=142ms TTL=50
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Reply from 208.113.226.219: bytes=32 time=140ms TTL=50
Request timed out.
Request timed out.
Reply from 208.113.226.219: bytes=32 time=89ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=85ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=87ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=86ms TTL=50
Request timed out.
Reply from 208.113.226.219: bytes=32 time=85ms TTL=50
Request timed out.

It’s almost like a comedy of errors at this point, first a DDOS attack and now a data center outage. I almost feel bad for the storm that’s come upon Dreamhost.

But things are back now.

Rather than notification as to the status of the system, I’m more interested in what they are doing to prevent this kind of thing from happening in the future. Right now that’s the more important question that they need to answer if they’re going to keep my business.

The ball is now in your court DH, please let us know how you plan to lock this stuff down. Doesn’t matter if it’s a blog post, email to me, or something in your silly little news letter. Just need something, these outages are the only thing that I hate about your service, everything else is phenomenal.

This is getting ridiculous…

FTP Issues AGAIN?!?!?!?!?!?!?

I was informed that my account had been infected with the script something blah blah blah and my site is deactivated so that the DH is intended to address the problems by removing the site and still threatening to disable my domain?

Still experiencing intermittent outages. My entire office building is looking at me like I’m some sort of asshole. Thanks guys. Does your uptime guarantee cover my unemployment when I get fired?

Was up. Now its down. Right in the middle of one final upload for a client. One upload = rent paid.

I was about to delete this, as the server kicked back on.. buttttt now its down again.

Ive been with you since 2007 dreamhost. I love you. Really I do. And no, I have no desire to find another host. etc.

But get this fixed! I got 4 mouths to feed! =p

WTF!!! is down again!!!!

So yea we’re down again?

Problems Again.+

Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5

“Still experiencing intermittent outages. My entire office building is looking at me like I’m some sort of asshole.”

@Kyle K hahah.. thats how I feel with everyone calling me wondering whats going on.

my site just went down again :(

SOME PEOPLE ARE TRYING TO WORK!!!!

Down again…HTTP ERROR: 504

Gateway TimeoutHTTP ERROR: 504

Gateway TimeoutHTTP ERROR: 504

Gateway Timeout

Ouch, this is bad, time to switch. Costing me $$$.

DOWN AGAIN.

Down again. I’ve been up and running for the last two hours, but down now. Dang. This just stinks.

Can you move us to the same place where is hosted this Dreamhoststatus ??? ¬¬ …

Not working again.

My clients are calling me wondering why my site is down, and the responses that I have the DH is as follows: Your site is infected! It must have been right after it was invaded DH?

But get this fixed! I got four mouths to feed! = p

Father, in Jesus name, please, help the DreamHost’s team to solve these problems that are distressed many users. You have all power across the world and in all thing, such as in the servers. Please, touch with your wisdom in these guys to fix all bugs. In Jesus name I pray and give thanks you for your help. Amen.

The problems have recurred. I can’t say I’m happy about this.

DOWN AGAIN. THIS IS KINDA FUN, ALREADY. CAN IT GET WORSE??

^
hahahha

Was up, now down again. This really blows.

Not working again. System is out aff

roflmao. thank god for this hoststatus page. If I couldnt read everyone else’s misery. Id lose my @#!@$@ mind.

“Father, in Jesus name, please, help the DreamHost’s team to solve these problems that are distressed many users.”

Where do we get a refund? This is absurd. It’s ruining my posts.

Update this fracking post, sites are down again

“Someone tripped over the plug. We’re trying to locate the outlet. Should be up soon” – Ralph

My website is down again I think. WTF. God damn. I wasn’t going to complain before, but for you guys to say things looked stable and to have it just go down right away again does NOT give me much confidence that you know what’s going on.

We pay the DH for this?

ok I’m down again during a client interview wwwooowww how embarrassing

UGGGGH BACK DOWN!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ARE YOU FUCKING SERIOUS!!! DREAMHOST IS A JOKE!!!! I DIDNT EVEN HAVE TIME TO DOWNLOAD EVERYTHING SO I CAN MOVE TO A NEW SERVER! I WANT MY MONEY BACK! Anyone know the legal options to recover some of the money they are loosing us????

Holy hell, and this is the 8th time or so today it’s down. How do I get a refund for the downtime???? I’ve prepaid for the year.

Going on break again.

Noooooooooo…. one more time????

I am back down again…

Stable??? Our sites are down again.

One site is up. The other missing in action. Please resolve ASAP. Thank you!

April 12th, 2012 at 3:22 pmbuythiscomputer Says:

Pay for a dedicated and waste your money !!!

http://discussion.dreamhost.com/thread-134721.html

I have a deadline to adhere to. I can’t work if my site is down. :|
Glad this is here and people are allowed to comment. I would be losing my mind right now if not for these status updates and comments.

All sites offline again … critical applications running… please help

Seriously questioning why I convinced my clients to go with Dreamhost. Both sites are still down, even though you’re saying it’s only affected one data center. PLEASE GET IT TOGETHER. Pretty offices… please make pretty servers work!

Where do we get a refund? This is absurd. 2!!!!

Is this a joke? Again? I want my money back. I can get better hosting in Albania.

Down here too, but complaining about it really makes them work faster to shut you up. Come on people. It is frustrating when you have 1M visits a year to have them see a blank page. But simply informing the staff is more productive than crying about it. I only wish my CloudFactor cache was complete…

Well thanks DH, waited all day to announce an event and pay for FB advertising. Just have you not have figured it out. I bet your service center isn’t taking calls right now is it – cowards.

@Val why don’t you use WAMP or XAMPP? It’s like having a server…and works great for days like this.

AGAIN?! Are you friggin’ kidding me?!

DH where is the solution for those who have paid for a dedicated hosting?

Down again! In case you dont get it yet… :)

Yup. Can confirm that it’s down for the 2nd time today.

Dammit. I hate shopping around for hosting. Please don’t make me. Aaauuuuuugh!

Sigh. I guess it’s like the old saying goes: When life throws whales at you, you die. Because whales are very large, and you’re probably not going to dodge having one thrown at you.

Wow, down again!

Of course hate filled rages on here are not going to get the servers up and running. But it sure feels good to sit here and read everyone elses happiness.

DOWN AGAIN !!!!!!!!!!!!!!!!!!!

it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous
it’s ridiculous

I sent support 3rd e-mail just in case they forgot about the customers.

Problems Again.+
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5

Up for a couple of hours and now back down, please please sort it out.

We’re trying to get a site online – is there any end in site???

Oh my God, I can’t believe it! Again?!?!?!?!

5:28 DOWN AGAIN WTF

Omg DreamHost! My patience is running out!!

I will have to find another host… sad!

My Sites are still down!!

“We are confident that this incident is resolved now. ” REALLY? DID YOU TRY TO OPEN A SITE?!

It is NOT resolved. Our site came up at 12noon and 3 hours later is down again!

Site of my client is out.

Confident? STILL DOWN!

This issue is *not* resolved – Irvine just went down again.

Bad timing for the last update Oscar. Nothing is working.

Bad timing for the last update Oscar. Nothing is working.

And the money paid? Will be returned?

I know this says it is resolved but our servers just went down again… are we sure this is resolved?

Update Apr 12, 3:25pm PDT: We are confident that this incident is resolved now. Distributed Denial of Service (DDoS) attacks triggered a instability of BGP routes to our peering partner. Now that we understand the source of the outage we have installed a DDoS mitigation appliance to address future DDoS attacks. We will also be announcing some further maintenance to our routing infrastructure which will compliment the current hardware and assume the role of managing our BGP routes. The addition of the DDoS solution will help prevent near term issues, but our commitment is to upgrade our networking infrastructure to provide you the stability you expect. Please keep an eye on this status post for future maintenance announcement. – Oscar

ALSO NOTE: They have changed the status to resolved in the past 5 minutes

This is not resolved. My sites stills down.

Bad timing for the last update Oscar. Nothing is working.
These guys do not know WHAT they do. Have a professional

I am sorry. As of 3:30 pm PST, and despite the update above, my site was working, but has gone down again.

Trying to stay calm relative to all the other rude comments here, but it is upsetting to see a status saying the incident hasn’t been resolved, right when issues are starting again

Maybe Resolved = It is 4:30 and time to go get a beer and maybe work on it tomorrow? :-P

I <3 you DreamHost! You're my heroes!

The update says everything is back up again, but my site is most definitely down…. Any new updates?

It’s unbelievable, it must be failing this shit dreamhost

Does anyone know how to get refunded for downtime? …because after 10 years on Dreamhost, they owe me a big fat check.

Troll.me has been down all day…wtf :(

Grrr the site is down again

Our site is not up as of ‘resolved’ 3:25PM status update.

Shame the DH!

My sites are still down too… “this incident” is NOT resolved!!

April 12th, 2012 at 3:33 pmNightmareHost Says:

NightmareHost!!!

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? V

My sites are still down.

seeing the same as others here, site is down again despite resolved notice (thank god for cloudflare keeping us at least presentable)

*Tag your it… but I didn’t want to play games…* Ok everyone lighten up…this is part of hosting, no matter where you go it will happen.

Site still down. Has been down since around 3pm PST.

Confident of what? We’re still down and we need to push a whole host of new features from a dev server to a production server! We need a hard response re resolution! Did you even test this?

Hello,

I am writing to inform you that we’ve had to disable the following
script(s) from running on your site(s) due to the script(s) being
compromised and running an outbound DDOS:

/home/…/funkneurotico.net.INFECTED

This is negatively impacting the network and overall server performance.
This issue needs to be addressed immediately. Please do not re-enable the
script by renaming or re-uploading from a backup as it will not fix the
issue. The script has been compromised and will most likely get
re-infected. Should you re-enabling the domain without fixing the
problem, may lead us to permanently disable the domain. If you have any
questions or concerns, please feel free to email me.

Thanks!
Oscar

As infected?

Mine’s not working.

Site is back down. I do not believe this is resolved.

Does cloudfare work good for ecommerce stores or just plain info sites?

April 12th, 2012 at 3:35 pmNightmareHost Says:

AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN? AGAIN?

Website came up and then went down again. WHEN ON EARTH will they go back up again?

Our site is not up as of ‘resolved’ 3:25PM status update.

My shared sites are all up. My vps is still offline and cant even ping it.

It’s 636p in NYC, 336p West coast,And I’m STILL DOWN!!!!!!! come back on a couple times, But STILL DOWN RIGHT NOW….PLEASE!!!!!

Without FTP access unfortunate…

lol @ Oscar, are you the head monkey there in data center?

My VPS is still down. Was back up a little earlier but is back down now. Please get this fixed!

Is this seriously marked as resolved right now? No sites are running what so ever. This really gives me confidence these people know what they’re doing….

FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!

Our site had been down off and on all day… When will this issue be resolved? Was an email sent out notifying us of this issue?

Celso, so you’re saying this is all YOUR fault?!

Have several client sites down. Help!

Er, it’s still down here in NYC.

Site down again! Issue not resolved! Please get it up again ASAP.

Hey PatrickD I say we blame Celso (Rule #2 – Always have a scapegoat waiting).. Just kidding Celso :)

WTF? Are they ignoring us? I haven’t got one single reply for the support tickets.

I was up for a few hours, but just went down again in NC…!!!

still down here in Europe…Zzzzzzzzzzzzz

I see that at 3:25 this issue was marked as resolved, yet I am still unable to access our site. How long will this take?

Looks like we are down again?
This is becoming all too common. What are we paying for?

shame!!

The website has been down all day and still isn’t back up.

seems to be down again already I’m afraid,

everything is down in Canada

It’s all CELSO’s Fault! Lets gang up on him and make him feel like #()!@#$!@$!

hahaha

As soon as you report this as resolved, our website goes down again. I am going to recommend moving our site to the director. This is not good.

I am down again. It is NOT NOT NOT resolved. You owe me $5,000.

still down here in Brazil…Zzzzzzzzzzzzz

All of my websites are still down, despite the resolution announcement. Is a delay to be expected?

April 12th, 2012 at 3:40 pmNightmareHost Says:

FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!

Mano, o nome do cara é OSCAR no máximo é troféu impressa!

this shit should be in a scrolling feed. that way I can laugh my ass off without hitting refresh every 2 seconds

Sites down again!!! This now has become a sad, sad joke!!! Supporter for years that has been professionally embarrassed by my choice of DH as a tech partner.

I have to agree with Steve on this one. If the incident is ‘resolved’ then explain why my websites are all still down. You guys are killin’ me here!

FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!

All of my sites are currently down residing on one VPS, second time this week, not good!

Definitely not solved, why the lie, I have 2 VPS, both down most of the day.

NOT RESOLVED, MY SITE IS STILL DOWN AND HAS BEEN SLOW FOR WEEKS!

Mine are still down as well – they were up for a while but now they’re gone again. I don’t think the issue is resolved yet.

OK, we’ĺl migrate to gandi.net

Update Apr 12, 3:25pm PDT: We are confident ???

SO IS IT THAT THIS ISSUE IS RESOLVED BUT THERES ANOTHER ONE?!!?!?!?!?!?!?!??!?!!?

Not resolved?? why do you say it is and it isn’t?

Yes, I’m still down as well. I’m really glad I sold my company on using Dreamhost

Haha. I mean this is a joke. I have only non-important sites on DreamHost but today they have been down all day and they are charging us a lot of money for their Full Moon Dedicated plan. It’s an embarrassment. I am done. Seriously. Even if I am lazy and don’t want to pay more, I’ll just move these non important sites to somewhere else or shut them down. Not worth the trouble this DreamHost crappy service is giving us.

I am confident it is not resolved

Guys, guys you are missing the point.
It’s all Celso’s fault!

Was the 3:25 update posted ironically? Resolved like hell! This is getting ridiculous. Today was supposed to be my site’s launch day and all I’ve been able to accomplish is wearing out my bloody F5 key.

I’m still down.

We’re still down, amigos. At least some sites are. Other dreamhosted sites I oversee are working fine.

I really hate seeing this happening to you guys. You’re as non-corporate as I’ve been able to find, and I really want you to keep going.

Wonder who’s out to get you? Hmmm….

SITES STILL DOWN!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

VPS PIECE OF SHIT!

Is anyone posting tickets through panel.dreamhost.com? Don’t think they are looking on this thread here anymore since they are so sure that it is resolved -_-

The status says resolved but my site and all sub domains are still down.

My site is STILL DOWN. THIS IS NOT RESOLVED.

My sites are still down, as well.

Either there is another issue or this one has not entirely been resolved.

looks like dreamhoststatus.com is hosted on linode.com…… heh……

Our site is down again. Agreed–it is not resolved. Filed yet another ticket.

Are any Dedicated server users being affected?

“We resolved the issue by taking down our irvine datacenter. Any websites hosted there will cease to exist. Thank you for your money.”

FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!
FIX IT BEFORE YOU MARK IT AS RESOLVED OSCAR!!!!!!!!!

Are you really “confident that this incident is resolved now?” Our website is still down and has been down for over 2 hours today and multiple hours in the past week or so. I do have an open support ticket with no response yet.

Definitely not resolved!

GUYS! Post tickets so they are informed, not sure if they are reading this tread since it is “resolved”

I have 3 Patricck

This is far reaching, ie. across the board. Im a dedicated host client. All 40+ of our sites are down.

April 12th, 2012 at 3:46 pmMark Erickson Says:

Not working as of 3:45 PST on 4/12/12. Argh!!!

How is DreamHost in business? I wish there was a way to sue for malpractice. I mean as a SEO, I can get sued for malpractice. WTF

ugh oh. hit F5 about 10 times in a row. no new comments. End of days is here. Its all over. Goodbye. world.

April 12th, 2012 at 3:47 pmBill Watters Says:

I just received a support response that indicates that not only the drive failed but also that the backups may also be bricked. So that I might “have to restore from offsite backups”. If that comes to pass, and that they don’t have a single viable backup going back months, then I’ll have lost clients, $, livelihood. They claim business class infrastructure, if we’re simply lost without a paddle, then next people I’m calling will be the press.

OK this is a new issue report is now a DDOS attack they said they installed mitigation against…just posted

http://www.dreamhoststatus.com/2012/04/12/network-issue-at-irvine-datacenter-2/

Sites have been down more than 6 hours. Coming back up for 10 minutes doesn’t count as being up and online. They don’t even have a phone number that I could call.

guys, there is a new status update on the status page. they are aware it is not working

Uh…my site is still down. This is good, I totally wanted to spend all day trying to find alternative solutions to this problem…

Not resolved. My site is still unreachable at 6:48PM EDT.

My site has been down pretty much all day and this is not acceptable! What is going on with your hosting, as this is a routine thing the past couple of months. I need to start looking elsewhere, as I had a bad experience with FatCow and someone recommended your service and it’s turning out to be even worse now.

As of 3:51 PM PST our sites are up.

After being down this morning our site was back up, only to go down again at 3:15 PST. Pingdom alerted us that it was back up 40 minutes later. How many more outages before we have to move hosting?

up again.

My site has been down pretty much all day and this is not acceptable!

@young linode.com , gandi.net?

Celso, You caused it! hahaha

My websites are back online now.

Was up again for a few moments. Now down again.

….and they are all back down again.

down again

Larry they are trying to find a culprit right hehe

last update report 3.25pm. Now 16.04pm down. Nothing to say??

Up now :)

Wow, Down Down Down. This obviously isn’t good and you know it isn’t when you have Godaddy calling you asking if you want to switch host providers!

Maybe it would be more time efficient and economically sound to just go back to pasting fliers up around town for your business rather than trying to host a website with DreamHost.

I’m still threatened BY Oscars?

Unfixed? Mine all died around 4:30pm PDT again. They were only back up for 4 or 5 hours =(

The sad thing here is that people are blaming DH for the work of hackers and then actively giving those hackers new targets by announcing/shilling for other hosts.

Incidentally, the people posting long lines of repeating text? They’re probably connected to the people really responsible, seeing as they clearly have 4-chan level discussion skills.

Site Down again.
Restored for a couple of minutes around 2.00 PST, but down again now

Up and down… ridiculous! Glad I moved my sites off to other hosts… but some of my clients are still on this… moving them off soon. What a ridiculous level of service. To the poster that said “all of them go down” or something like that, not true. Other hosts that I mentioned (and got my post deleted) before haven’t had downtime in years. Maybe a minute to 5 minutes in maintenance, but hardly the same.

Really people, this level of service isn’t normal.

April 12th, 2012 at 6:14 pmNigel J. Terry Says:

My sites have just gone down again

DOWN AGAIN

We’ve been up and stable for a while now, though a few occasional outages. nevertheless, Thank you DH for addressing this issue. A DDoS is never good and its great to see you guys jump to our saving grace. I saw one of the DDoS hit one of my sites, massive flood of nearly 2K individual requests in one run. It was a nervous time, knowing it was going on and I had no way to stop it. Thanks, and on behalf of those who just saw the outage and needed to blame you guys, I thank you for helping them too :)

It is not clear if Dreamhost thechnicians, is a complete idiots, or a big asshole!

my sweethearts birthday gift: http://www.iamsushma.com
is working now
thanks dreamhost.

Jyoti,

Guess what the sites are down again… third time in less than 7 days

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Minor switch issue (RESOLVED)
6:23 pm

Minor switch issue (RESOLVED)

Posted (April 9th, 2012 at 6:23 pm PST) by Ralph

We are looking into a switch issue that is affecting two racks at one of our data centers. The list of machines is listed below.

andromeda, antila, apus, aquarius, aquila, ara, auriga, bootes, bullhead,
caelum, camelopardalis, cancer, canis, capricornus, carina, cassiopeia,
centaurus, cepheus, chamaeleon, circinus, columba, crux, cygnus, delphinus,
dogfish, dorado, draco, equuleus, eridanus, fornax, gemini, grus, horologium,
indus, lacerta, lepus, megamouth, sandtiger

We are addressing the issue quickly and will provide an update as soon as we have more information.

Update Apr 9, 7:20pm PDT: This issue is resolved now – switching ports resolved the issue. We will continue to monitor the switch for any further signs of instability and, in addition to that have begun the process of replacing the hardware involved to preclude any future incidents.

Update Apr 9, 7:10pm PDT: We’ve migrated the uplink on the affected switch to another port and some connectivity has been restored. However, we are still seeing minor packet loss to various IPs on that switch so there still remains an outstanding service degradation issue. Our Network and Systems admins are still collaborating on the issue and we’ll post more information as it becomes available.

Update Apr 9, 6:49pm PDT: Switch is up and we’ve confirmed management level connectivity however it is stll unable to get to the outside world. We are continuing to troubleshoot the switch config as well as its uplinks to the cores

Update Apr 9, 6:43pm PDT: We have rebooted the switch and are still investigating this issue. We will update this post regularly till resolution.

This entry was posted on Monday, April 9th, 2012 at 6:23 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

50 Responses to “Minor switch issue (RESOLVED)”

All my websites are down!

@Marisol

What kind of response do you expect with that comment? “Oh, sorry, we’ll just turn everything back on. We didn’t know you were using that. Sorry.”

I’m just happy they posted the status.

Well, that’s it. Loyal customer since 1999, but I’m in the midst of sending out my portfolio for art directors and this is the second outage I’ve had in about a week. I work in games, and a down site is a totally lost opportunity.

Leaving Dreamhost, unfortunately.

Outages are becoming almost a weekly event. I hope you have plans to get things under control. We’re looking for alternatives for our 50+ sites hosted through DreamHost.

man, ya’ll just can’t catch a break eh?

two of my sites are down (30+mins); but does anyone care at DH – nope…I don’t believe they do…I wish it were otherwise…

I wonder what’s going to break next;

*here’s my giveaway million dollar idea: have a website with a dreamhost downtime/crash pool

Will you guys have it up soon. Oh and this sucks btw.

@Justin

What a loser response..

Not sure what’s happening with DH. They have a quite a good track record in the past. Enough that I left in ’02 and came back in ’07. However things have been a little sloppy recently. My last ticket was not even opened in 17 hours until I retracted it. And now it seems like downtime is becoming somewhat of a frequent event. (too frequent for it’s time & place anyway).

Not trying to find someone or something to point a finger at, but it seems to me that things have been moving in the wrong direction at DH since they hired their first CEO.

‘Minor’ issue involving 38 machines? What’s a major issue?

This is total BS!!! This is the second time I’ve been down in less than a month. I’m launching a new site as a result of DH being down I can’t test. It’s a real pain in the ass to change hosts but at this rate the down time is beginning to become a regular issue and an interference in running a business.

Gods Speed Dreamhost. Get it back up and running.

very disappointed, also. 11 years and no real problems until the last few months. i know these things are hard to avoid sometimes, but better high availability might be nice. single points of failure = fail! =)

Well, my websites are back.

Not again. Really bad timing, as usual. You’re killing me, Dreamhost.

My sites were down for a short period of time again today, second day in a row that I came into the office unable to manager the sites.

Also is there an ETA on when we will get live chat support back?

our website, database and ftp is currently down. Any ETA on the cause and fix?

my websites are down, despite the problem being reported as “resolved”. What does “resolved” mean again?

our website, database is currently down. Again, Again, Again…

this is now beyond BS…Dreamhost hasn’t even acknowledge that there is a problem…looks like its time to move on…after 4 years and no problem the past three months has been a real inconvenience. It wouldn’t be as bad if they at least had the respect of their client and informed us of what is going on. Apparently this has been afew hours and they still have “resolved” posted ….that’s a load of crap!

Sites down for 3 hours now again today. having a database connection error and I can’t even get into the backend to check on it or pull the database so the site can run on our mirror. To think I was considering dreamhost as the mirror to dreamhost is now obviously questionable on my part. Glad I never did that, but still this is getting a bit too much to deal with because there is no one to talk to. I purchased support for this account, and when i did, at least the online help chat was running. That has been taken down along with my confidence in dreamhost knowing what their own issues are….

my websites are down!

My site has been down for 6 hours…I did speak to someone at Dreamhost but my dog could have given me more information. We publish a newsletter e very Wednesday and have since 2005..the information needed to put the newsletter together is on the site. Dreamhost hasn’t even had the professionalism to keep us in the loop. I’m grateful I don’t have an e-commerce site. This is pure BS I have no choice but to change servers…this is the 4th time in 2 months that we’ve been down.

Can anyone recommend a good host?

I wish they would take the word “RESOLVED” off this page it is only making me angrier!

so, my site is on Andromeda. Here’s the feedback from my support page in my panel…

—-

Current Outage: Hardware failure on hosting machine andromeda (updated 42 mins 56 secs ago)
Hello,
The NFS mount from the backup machine to the new hardware has been created. This allows your sites to load as the restore is running.
There are a few apache instances that are failing to start. This would explain why your site remains offline. Our admins are diligently working towards getting the apache servers online.
We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact our support team.
Please note, this is regarding the shared web server �andromeda� and no other services (email, MySQL) or servers are affected.
Thanks!
The Happy DreamHost Server Fixing Team

3 days with all my sites slow and down…
We need a new host, Dreamhost is not the same as 5 years ago.

My server is Becrux

I’m moving my site Thursday ….I’ve worked too long and too hard to have a service (I wish they knew the definition of service) put it all in the toilet. They should have a back up plan to solve situations like this that may come up…right now I feel like I’m in the Titanic ….I’m not going to go down with the ship..

Going on 8 hours of no site…

The Happy DreamHost Server Fixing Team I’m really glad to hear that they are HAPPY!!! Because I’m certainly not.

come on fix this now!!
Sites have been on and off for last 48hours plus now mail server is playing up as well.
I have new clients whom I have referred to dreamhost and this is looking great after just a week….reflects on me more than you guys as I am copping the grief.
am on carina… definitely not resolved.

Here’s my admittedly inexpert guess with what’s happening at DH:

They used to be pretty good, and because of that well-deserved reputation, they’ve expanded a heck of a lot. Now they can’t keep up with the expansion. Maybe they’re so focused on getting those new customers that they can’t or won’t put the time into keep the infrastructure suited to that expansion.

Whatever the reason, I’ve been meaning to bail as well and am looking at Media Layer or Media Temple.

Come on DH, some of us are being patient, but it’s day two and my websites are a mess!

April 11th, 2012 at 8:07 amJonathan Tonge Says:

Last week I complained about ten times in the April 02 Status Issue. I was back up on April 4th at 5:44. Now I’m having intermittent connection issues both on front and back end. Every other site on the web works but mine.

What am I suppose to do with you guys? Is it acceptable to be down 1 or 2 days a week? Is that OK with you? Because I think it is absolutely horrendous. I think noone, and I mean noone, in this highly competitive marketplace that you exist in, would sign up with DreamHost if they were aware of the issues you are having.

April 11th, 2012 at 8:09 amJonathan Tonge Says:

I think this is about the twentieth time this year that you’ve been down. You sign people into contracts but do those contracts stipulate an uptime quota? I’m going to look into this and flag it to a big lawyer. I’m sure a class action lawsuit would be profitable based on all these issues you are having and the millions of potential customers that would want to be refunded.

April 11th, 2012 at 8:13 amJonathan Tonge Says:

Even if you don’t stipulate it, a court would use a percentage that would seem ‘reasonable’. I can’t imagine that being less than 98%, if not 99%+. Other companies are offering 99.9% uptime based on third-party assessments. This year I’m experiencing somewhere between 85-90% uptime with DreamHost. It’s absolutely unacceptable.

I transferred my web hosting service last week after the last major outage and I’m happy that my site is UP. I think all the downtime has affected my website traffic and now I need to slowly bring it back up. Sorry Dreamhost. Your response and support team is great and I was happy with you for about 5 years, but it was time to leave. I’m with Tigertech now.
elaine

And now…

PHP Parse Error: syntax error, unexpected $end

Just to say; it’s fantastic!!!

My site is down again. AGAIN. 3 times since the 7th. This is really a problem.

Websites are down again.

My site’s down again. Apparently. Also 3rd time since the 7th.

MY SITE IS DOWN AGAIN!!!!
ALWAYS THE SAME PROBLEM.

Down again, no updates.

Not happy, again. It’s a real bitch to move my 35+ sites, but I will NEVER recommend Dreamhost to anyone, ever.

Seriously what in the hell is going on! My site is down again 2nd time within a few days?

I’m down again.

Dreamhost I really enjoyed your service to date, and I hate people that bitch on your status blog posts. BUT

This is ridiculous, twice a massive outage during business hours IN THE SAME WEEK?!

We’re going to have to talk, or I’m going to have to consider taking my business elsewhere and getting a refund from you.

Please help!

My site http://www.seguroviagem.srv.br is down again, crucial applications running on this domain.

Please help! People are dying here!

Looks like we’re down again too.. Dedicated server here, hosted in the Irvine datacenter.

Why isn’t there a blog post about today’s issue? Second downtime within a week.

My Chamber website is down right now. Is this related to this issue, or is there something else going on?

Site is down. No update. What gives?

I’m helping a friend with their site BUT this is why MY business left DH a year ago, for Hostmonster.

Dreamhost thechnicians is a complete idiots, or a big asshole?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Irvine Datacenter Connection Issues (RESOLVED)
8:15 am

Irvine Datacenter Connection Issues (RESOLVED)

Posted (April 9th, 2012 at 8:15 am PST) by Justin@DH

We are currently experiencing issues with our Irvine datacenter caused by a DDoS attack. We have our network team looking into it and are working to get service restored as soon as possible. As soon as we have further information we will be updating this post.

Update Apr 9, 9:21am PDT:We are happy to report that all services were restored around 8:40am. We have been spot-checking various services to confirm that the network was fully functional. Our network engineers will continue to actively monitor the stability of our network. We apologize for any inconvenience this has caused and appreciate your continued patience. If you happen to notice any issues still, please contact support and we would be happy to assist you. – Justin

Update Apr 9, 9:00am PDT:Most services are restored, although there still could be one network that is affected that we are looking into. We are hope to have all services restored shortly. – Justin

Update Apr 9, 8:40am PDT:We are seeing some services restored in the irvine datacenter, but there still remain a few network ranges that are not reachable. We are continuing our work on a datacenter-wide resolution. – Justin

Update Apr 9, 8:20am PDT:The cause of the issue is being narrowed down. The issue is still causing downtime for customers in our irvine datacenter. We have our entire team working on the issue. – Justin

This entry was posted on Monday, April 9th, 2012 at 8:15 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

169 Responses to “Irvine Datacenter Connection Issues (RESOLVED)”

please, please, help us!!!

COMMEEE ONN YOU GUYSSS !!!!
You are not serious\!@#!@$!@%!@%1

My sites are all down. My business has stopped!

And once again, we’ll get some stupid email saying sorry and not the slightest bit of recourse. It’s ok that your down times cost your customers millions of dollars. I cannot wait until we find a new home.

Second outage for us in ONE MONTH! Not cool at all.

Nunki server offline!

Will you order us some pizza while we wait?

The same problem again!… wtf with Irvine datacenter…!!!!!!!!!!

Does this effect email too? I cant send emails through the gmail account. Does anyone know about that?

I was debating whether to go with media temple or dreamhost (LAST NIGHT!!!) is that mad or what?

OMG!!!!!
i’m currently losing money now !!!!!!!

do something about it !@! @!@!@!@!@

DreamHost Fail!!!!!!! Dedicate Server!!!

Website Currently Unavailable
The website you are trying to access is currently unavailable. Please try again at a later time.

If you are the site owner, review the following resource to help resolve the issue.

© 2011 CloudFlare, Inc.

MGM SITES ARE DOWN !!!!!

stop playing and behave seriously.
one problem after another, I get up in the morning praying they do not have a problem.

CloudFlare? NEVER USE THIS!

Please get us back up and running

Our website is down too! are you kidding me? this isnt the first time with the Irvine datacenter!!!!!!!

I think this is the last straw for me. I can’t believe they don’t have diverse routes and multiple providers into a Network Center.

Stop playing and behave seriously.
one problem after another, I get up in the morning praying they do not have a problem.

All websites on our cloud are down AS WELL AS our dedicated server. The 2nd complete shutdown in 30 days, the last one lasted for about 8 hours and cost us 4 clients.

Our website has been down for 30 minutes during peak hours. This is not acceptable. Dreamhost, please get your act together, or we will have to consider using a different provider. Many thanks.

Pleaseeeeeeee!!!!!!! help us!!!!!!1

How do I know in fact if my site being down is because of this issue?

rukbat server down

Our Website is down as well. We are on Miriam.

Well, great. Less than two hours after I create an account here, the service goes down. What a croc.

come oooooooooooooooooon!

This happens way too often. I’ll be canceling my account and moving my domains this week.

Looks like something failed with the router?

The 2nd complete shutdown in 30 days!!!!
This is a dedicated plan?

I wont be surprised if some cable is unplugged :)

My site is down too! It’s not the first time this year.. Come on guys..

And you guys can kiss my ass with your “thanks for your patience” crap and cute newsletters. We don’t want jokes. We want uptime. Cute and clever newsletters don’t help us.

ps61126 Down and out…

So we just published a bunch of high-profile game previews and sent them to our PR reps. And we’re running contests. This is the 2nd outage in less than a month. The last one was almost 8 hours. If we aren’t given a serious discount on our hosting bill, we’re gone. We don’t pay for a dedicated server to have these constant problems, Dreamhost.

Nunki is a victim once again. Last time we had an outage, it lasted 8 hours and this was just a couple weeks ago. What is happening guys!

We just had made a very important post and the site goes down… this could not have been a better time!

Come on Dreamhost. I never once had this problem with my previous hosting company. Should I go back?

Of course…. Dreamhost sucks

you may need to remove this from your site “DreamHost manages its own network, so that your site never skips a beat. DreamHost’s network is backed by a 100% uptime guarantee policy”

Yooooooo, Is this what this year is gonna be like?? Every other month we can expect some days just not to be on, I really am starting to think we are waisting our $!!!! And the online chat has been down for months, Not what we signed u for -_-

Our site is down too…

UNACCEPTABLE

April 9th, 2012 at 8:24 amChristopher M Says:

This was the last straw. I’m moving my sites off dreamhost as soon as possible.

For people saying this is the second downtime for dedicated servers in a month (last one 8 hours) when did the last outage occur?

This is not good
my sites are down too. Lets see how much time they take to resolve the issue

For me, for a $6/mo host, this is to be expected. I hope those that are flaming dreamhost are on a production-ready hosting plan here… Then you have a right to complain.

I had big cross-promotional email campaign for my site (hosted on DreamHost) this morning that went out to 15,000 subscribers. So much for that. Thanks Dreamhost.

DreamHost Fail!!!!!!! Dedicate Server!!!
PLEASE PLEASE

How come we don’t get compensated for these down times? It’s not fair that we lose money and you guys get to keep all our money. We all should be getting some sort of credit.

Patience people. We understand that these things happen. dreamhost is an excellent hosting.

whoops

all my vps are down.

hope all is ok

regards

I really love using DH, but these stressful moments… the puzzled phone calls from clients when things don’t work…

I guess I’ll have to move critical hosting elsewhere and keep DH as my technical playground.

Fired! DH no longer deserves our business, or anyone else’s business for that matter.

Glad I’m not trying to make money on this yet. But I’ve flushed out one bad bet for a host. I don’t even know If my server is in Irvine, and don’t know how to check. But I know that I am currently down.

Waiting patiently for the sites to come back up. Thanks Dreamhost team for all of your hard work. Thanks also for catching the issue quickly. Please update as soon as possible when service is restored. My sites are on scheat.

DreamHost is a joke. If you complain more than a couple of times, they ask you to shut up and leave. That’s what happened to us so we decided to just shut up. Moving too many sites does not make sense for us at this point. For a dedicated web host, their service is abysmal

VPS DOWN!!!!!!!!!!!!!!!

It’s just frustrating that some of us pay quite a bit of money for dedicated hosting, and yet we’re still subject to these kinds of mass outages. I know no one can absolutely guarantee 100% uptime, but these long outages cost real people (like me) money.

…and why is Live chat support never online anymore?

This outage happened literally after I sent a marketing campaign that cost us hundred of dollars, which will result in thousand of dollars in lost opportunity cost. I pay extra for the call back support and still nothing!

Since becoming a customer 16 days ago we have experienced 3 serious issues! This is a company that seems to have the resources available to properly service their network of customers!

Guys:

Don´t complain. It happens to every one and every business. You all better install some protections like: CloudFlare (That is a free application to dream host) and you will be protected.

I didn’t even get a cute email telling me it as down. I had to wait for a client to tell me!

Correction: DOES NOT HAVE THE RESOURCES AVAILABLE TO SERVICE THEIR CUSTOMERS!

Not cool dudes!

We just promoted the hell out of my podcast for the past 3 days at a major convention and now Rana is down and my site with it. 5th outtage for me in 3 months, very much not within the 99% up time promise. Would love some idea of what happened, why, and how long till it is not an issue again

Sh*t happens, but unfortunately, Sh*t is happening too much frequently, though.

30 minutes of downtime and counting right in the middle of the Monday morning rush. This is a problem, guys. We count on you for uptime – where’s the redundancy??

Here’s your 100% Uptime Guarantee, to jog your memories of what you promised us. Looks like this is going to cost you an awful lot of cash (in addition to customers, I’m sure).

“DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!”

Dreamhost has messed things up enormously for us over the past week. My hunt for a new host begins now.

This is too freaky! this never happened to my site BEFORE i installed Cloudflare! Is this a cloudfare integration error? please restore my site it’s been down for the past half hour – right at the time where i normally update it. argh!!

honestly, DH does not perform very well since last year, many downtimes, and recently, no seats aailabel in live chat…

It has been down for almost an hour now.

Everyone open support tickets to receive your 1 day credit:

“We’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem.”

I was just trying to show a client their new website. This makes me look really bad.

These outages are now too frequent to justify continuing to do business with Dreamhost. It is time to find a new service provider.

WOW… So much for my productivity….. !!!!!HELP

It’s pretty hard to run a breaking news website using DreamHost if the site is going to be down when we need it most. It’s not like we can tell the news to wait until our site is up and running. We just started using DreamHost; is this going to be a persistent problem?

We’re on the Today Show this morning! Went down right as the east coast viewers were getting going. I’m gonna cry!!

@Jason

Exactly. The chat is never live for us. Never. I have tried it and it is absolutely never on. Not in the morning, afternoon, night, midnight, 4 AM, 6 AM … it’s a scam. It’s despicable that DreamHost has a chat system that does not work for dedicated host users. I have asked them this specifically but DreamHost just does not respond. :(

maybe you guys should consider changing datacenters … i have been with dh for over 10 years with a different account .. the sites on that account almost never have issues … this one that was created last yesr however always seems like we’re down or having some sort of issue and often i see its datacenter connectivity issues

Looking for a new host unless I am compensated for the lost business.

2 times in less than 2 weeks, not acceptable.

down again

“The cause of the issue is being narrowed down. The issue is still causing downtime for customers in our irvine datacenter. We have our entire team working on the issue.”

Folks, this is bad. This is REALLY bad. This sounds like it could be a day-long (or longer) outage to me.

Website Down. Urgent!!!!!!!!!!!!

I hope this is resolved soon, and the underlying cause fixed. Please keep us updated.

WHO THE HELL IS IRVING AND I DON’T CARE ABOUT HIM I CARE ABOUT DREAMHOSTING WORKING AT A CRITICAL HR OF BUSINESS AND EMAILS … COME ON GUYS I PAY WAY TOO MUCH FOR THIS TO HAPPEN!!!!! RIGHT NOW I BETTER HOPE I’M DREAMING THAT IS HOST COMPANY IF MESSING WITH ME?!????!!! FIX IT NOW! AND LEAVE IRVING ALONE OR TELL THEM GET A POWER SUPPLY THAT WORKS OMFFFGGG PEOPLE!!!! DO YOUR JOBS!!!!!

Not again… I wish Dreamhost could automatically re-route to a working datacenter when these outages occur.

america people have fast word SCAM …………………

Yay! Over 60 customer sites down plus multiple company sites.

DreamHost = NightmareHost.

All of my websites are down as well.

GOOD BACK THANKSS

“Will you order us some pizza while we wait?” hahahah xD

Looks like my sites are coming back online.

Thanks site started working now.

Seriously…

This is Monday Morning traffic and you guys are DOWN again… How is this even possible?

Hostgator is looking better and better… Especially since having my site down is a weekly event!

Dreamhost is starting to suck… hard.

Dreamhost…could you do me a favor? Remind me again, WHY exactly I’m paying you for a horribly inferior web hosting service? So far, the promised/projected figure of 99.9% uptime is WAY off. This is COMPLETELY unacceptable!

I have a dedicated server it was supposed to be safe and have no downtime. I am paying for chat support and the support is down ALL the time… what is wrong?

rubkat back on!

Now it works. Thank You Dreamhost.

Cheers from Brazil.

While I don’t claim the understand the nuances of Dreamhost’s operation, considering that failures are sometimes inevitable, I guess what I’m confused by is how there doesn’t seem to be a backup that’s kicking in to keep our sites online.

And just after typing that, my sites (on syrma) are all back up :)

Website came back 3 minutes before.

Back up – whew!

“Update Apr 9, 8:20am PDT:The cause of the issue is being narrowed down. The issue is still causing downtime for customers in our irvine datacenter. We have our entire team working on the issue. – Justin”

– I mean this in the best of ways… how is that a helpful update in any way? It would be nice to have just a few more details and maybe more of a timeline so that people can put into act a contingency plan or enact a management process that isn’t going to be wasted. Even an indication if we can expect to wait a short time or a long time roughly in hours or minutes so that a risk management plan can be implemented….. just a suggestion.

All my sites down too. Can’t be upset about this, I’m too cheap to pay more for hosting.

Everyone who is mad: settle down or fork out the dough to get a more reliable service. I don’t know of any other provider that offers the dynamic environment provided by Dreamhost at the same price. If you know of one, I’d be more than interest to know who.

If you don’t have suggestions that improve everyone’s experience, whine into your pillow and not on this blog. No one wants to hear it.

Dedicated Server seems to be back online. Thanks for the hard work, but unfortunately we’re going to have to part, DreamHost. You’ve been down way too often, and we do not see things getting better at ALL.

i agree mike.. come on.. its very cheap!

regards!

Okay, just 20 minutes of downtime. I think if any provider can sort shit out within one hour it is within the limit. Shit happens. Thanks for sorting this out swiftly.

onlymidia.com website down, Urgent!!!!!!!!!!!

I’m back up already.

We’re back … I don’t know how you lot can get so would-up over 25 minutes of downtime… it doesn’t happen every day.
I suggest you all take a chill-pill :-)

cheers from China.
well, it is nice to have a place to curse then cheer!

all working!

yes still down

jajjaja chill pill

We’re back, But guys this has been Too many times, And get your online chat back, or we won’t feel comfortable staying 0_0

Sites came back online and then went down again.

Our websites are up again. Hopefully they will stay that way.

Sites are back up!

Can you believe that they will only credit me for downtime from the time I posted a Trouble Ticket, not from when the system went down!!! How’s that for great service?

Let’s understand this. The service is down, they acknowledge that it is down, my client is on the phone BUT in order to get limited compensation that hardly makes up for the loss of business, I have to hang up on my client and open a Trouble Ticket for a problem that is already acknowledged!!!

Can someone explain how that is customer service??

When did DH become such a disappointment?. CONSTANT outages and issues!! Why do we pay you???

April 9th, 2012 at 8:56 amShawn Gossman Says:

My third party downtime reporting service reported 50 minutes of downtime. Now while I understand there will be downtime here and there, I am still paying $120 a month to you all for dedicated server, (ds2032) I would love to be credited for that 50 minutes if possible? Even if its just a few dollars, these days, I don’t like wasting even a few dollars :D

Hey guys! please, please, help us!!! my site is critical!.. DOWN!! i can’t believe this… HELP PLEASE!!!! this is terrible!!!

April 9th, 2012 at 8:57 amMoving to Media Temple Says:

@OW Says: The sites I’ve moved to Media Temple haven’t had so much as a second of downtime in several months. The sites I’ve left with DH have had outages weekly. I’m moving all my sites now.

A DDOS taking down a server is understandable, but the core network infrastructure? Jesus Christ guys, get some better networking hardware & engineers.

Frequent downtime every week, have lost at least $1,000-$3,000 in sales from the downtime over the last few weeks, just happily gave my money to PHPFog, switching now.

@Shawn that’s around $0.13 — not a few dollars.

My services are restored as far as I can tell. DDoS’ing an entire datacenter is unacceptable, but glad you guys were able to resolve (at least mine) the issues fairly quickly as opposed to hours and hours of downtime.

It’s all back up and hope it stays up forever.

Thank you! guys i hope that this ever happen again, thanks :)

April 9th, 2012 at 9:12 amShawn Gossman Says:

@Christan, then 13 cents credited it is ;)

Help… How long time more??? My customers was nervous…….

Our site is back up after almost 1 hour of downtime. Thanks for getting it back up, but please don’t let this kind of thing happen so often.

My site is still down……..

One of my sites is still down… This is completely unacceptable.

Our website is still down and people are freaking out.

@Ryan:

> So far, the promised/projected figure of 99.9% uptime is WAY off.

Why? 365*24=8760 … If DreamHost promises 99.9% uptime that means 8.7 hours of downtime per year.
If you don’t want that you should choose someone promising 99.99% uptime (maximal 50 minutes of downtime per year).

My site and VPS is still down… 2 weeks of failures :(

People, people!

This sort of things happens. Regardless of which host you go to, on occasion, somebody has the intent to hack a server for what ever needs, much like this occasion. I think it’s absolutely ridiculous that you all cry like little babies about some technical difficulties. If you went any where else, you’d experience all sorts of problems. This is the inevitability of running servers. DreamHost offers great prices for the number of facilities they get, and I am sure they work ridiculously hard to keep things running smoothly. So keep it calm.

If you want to attempt to keep things running smoothly yourself, go buy a physical server and host it yourself.

DreamHost is doing a fantastic job. Keep it up guys.

Let me add to this: It’s a DDoS attack. It wasn’t due to network hardware. There’s only so much they can do at a datacenter to stop those things. You can minimize the attacks, but if the Internet criminals are intent on getting their way, they will. Ask the cyberexperts who work for the U.S. government.

Yes, I grant having that happen on a Monday “drive time” isn’t helpful if you’re using your sites for business.But stuff happens.

Peace,
Gene

i have two sites on this host and have two weeks not working on server miaplacidus some imes going to turn back but again down one site destroyed my database and need to build from dhe start ! whats happen?

Host Not Configured to Serve Web Traffic
You’ve requested a page on a website that is on the CloudFlare network. Unfortunately, the host web server does not appear to be configured to accept web traffic or serve web traffic. If you are the owner of this website, you should login to CloudFlare and change the DNS records for to resolve to a server configured to serve web traffic.

If you continue to experience this error, you may contact support. Please reference the following information in your message:

Timestamp: Mon, 09 Apr 2012 09:56:43 -0700

Your IP address: 187.8.25.218

Error reference number: 1004

Server ID: FL_17F5

Process ID: PID_1333990603.515-1-1504430264

User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24

@ Simon Schick

I have no problem with 99.9% downtime. The problem that I do have is that over the course of the past year, I’ve had a full day and a half of downtime, which is WAY more than the 8.7 hrs/year promised. I would simply like a more stable backup system in place. Most hosting providers have backup servers or backup server farms, meaning that if one or multiple servers go down, there are other ones that kick in. If a whole server farm goes down, another location kicks in, and loads backups. This significantly limits downtime, at a minimal cost to Dreamhost. That is not an unrealistic expectation. It’s actually pretty common business practices. Just look at any ISP web hosting services.

For all people crying about the Dreamhost services, please, get out and pay for a 99,999% uptime datacenter. If you lost money, ok, use it right now to pay for a “decent” datacenter that you like.

I hate to read people crying like babies and paying few bucks. If your “service” is so important for all Internet (yes, we can’t leave without it, right?), look what Google do: make YOUR datacenter.

Stupid!

PS: Thank you Dreamhost. These people don’t know what is a real bad datacenter.

Greetings from Brazil

I’d also like to note that it IS a big deal if you’re hosting a small e-commerce site, that can’t afford a dedicated server. If their site goes down, all business stops. With a small business, that can mean a full catastrophic demise of said business. How do you think a physical storefront would fare if their doors were just randomly closed for a day or even an hour during regular business hours? Like it or not, downtime IS a big deal.

I find it strange that many people here are losing “a ton of $” while their $10/month server is down for 20 minutes…? If I had enough traffic and revenue flowing through my site that 20 minutes of downtime ruined my life, I think I would go elsewhere for hosting…

Maybe I’m not the norm, but we’re paying near $200 / month for a dedicated server, and our business is totally reliant on internet traffic, so when the site’s down for an hour, during peak purchasing time, it’s a big deal. Currently looking elsewhere for Cloud storage.

We pay top by DreamHost service and we have this garbage! The problem STILL has not been resolved, the support does not respond and pay for CHAT which he is no longer used. False advertising. Regrettable.

R: Maybe I’m not the norm, but we’re paying near $200 / month for a dedicated server, and our business is totally reliant on internet traffic, so when the site’s down for an hour, during peak purchasing time, it’s a big deal. Currently looking elsewhere for Cloud storage.

Ken, me too… more than 100$ each month a the last 2 weeks with important failures. It’s an awful situation.

Caste, Do not say bullshit!

@Paulino Como você é baba ovo dessa bosta de empresa, cala boca idiota!

I am still locked out, tried to contact support and got a response that I am not in their records as a current customer…. what is up!!

Site is still down. No response from tech support.

all my sites are down right now. wtf?

My sites, hosted on cairo, are down. I don’t know if this is part of the same Irvine problem or not.

Please let me know ASAP what’s going on. My customers are calling.

I’m down too and have been all morning. I just signed up for CloudFlare, which DH assured me would solve these problems like magic. This comes on the heels of a major hacking just a couple of months ago, which so thoroughly corrupted my website that I threw the whole thing out, changed all the codes, and tried to rebuilt it from scratch. All is rebuilt except the e-commerce pages, which I haven’t been able to get back the way they were. I was impressed with DH when I switched to them last summer, but not at all now.

The top of this page proclaims: Irvine Datacenter Connection Issues (RESOLVED) Guess they don’t even know what’s happening in their own system, because nothing has been resolved for me. It’s now 1 PM Mountain Time and my website has been out since around 9 AM. Pretty bad service.

We’ve been down for several hours.
Have filed a support ticket but have heard nothing.
Would greatly appreciate some feedback.

My site is still not functioning properly (if at all). This is quite a disaster.

yes, this is so frustrating. No phone line for help, no posting that says they are still working on this, only a RESOLVED note on their blog but obviously it is not resolved. It helps to see that I am not the only one still out of service. At least I know its not my computer.

DDOS on DreamHost? LoL … you forgot to activate your firewall :P ?

Issues:
All my websites are ON 100% working but,
SSH / SFTP are dead 100% No Responding!!

Guys, sorry. Actually I am a troll.

OMG, OMG, OMG… :(

Goodbye money!

Business toped. :(

WTF!!!

Still no email… anybody else ready to fire Dream Host?

Will our websites be coming back up any time soon? We’re serving multiple sites, and none of them work.

Guys. Relax. It happens, everywhere.
I had problems with Amazon (with good company pheraps), italian main hosting company (Aruba, down for almost an entire day)… It just happens. Emailing customers for issues? With what purpose, if the email goes down with the servers?
We experienced 6 hours down since 2005 and pricing/services are just incomparable with all the other providers we have.
If you need 24×7 reliability, you’ll have to put up some cloudflare, an alert service (like 24×7…), a nice backup server somewhere else and maybe some more DNSses.
And you’ll have to pay for it.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Intermittent network issues affecting some customers (RESOLVED)
3:43 am

Intermittent network issues affecting some customers (RESOLVED)

Posted (April 7th, 2012 at 3:43 am PST) by patrickm

Due to issues caused by maintenance being performed by one of our transit partners, an upstream link we have, become unstable. This forced us to have to shut down the link for the time being. We will re-enable it once the maintenance is complete.

Update Apr 7, 6:45am PDT: The maintenance has ended, the connection was re-enabled and been running problem free for 1.5 hours. -JJ G

Services that were disrupted by this outage are now back online. If you are experiencing any issues they are likely unrelated. Please contact support and we will be happy to help! -Patrick

This entry was posted on Saturday, April 7th, 2012 at 3:43 am and is filed under General Outages, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

65 Responses to “Intermittent network issues affecting some customers (RESOLVED)”

All my sites are down. This is really bad.
My customers are screaming at me …

All my sites are down too :(((

April 7th, 2012 at 3:53 amBrett Flannigan Says:

I noticed all my sites along with panel.dreamhost.com were inaccessible. I then checked on downforeveryoneorjustme.com and it was listing everything as up, and after that I checked again and everything was then accessible to me.
Now everything is again inaccessible to me while downforeveryoneorjustme.com is still saying everything is up.

So… yeah, intermittent indeed and not affecting every person trying to view the sites.

I can’t access anything, and of course it went down just as I was in the middle of some serious updates. This is seeming like a weekly occurrence now. When is dreamhost going to get it together? EVER? What are we paying for here??? FREE hosting services have better up time than this. Ugh.

April 7th, 2012 at 5:18 amUnhappyCamper Says:

All my sites are down. There goes my revenue for today.

Intermittent problems? My site’s either been down all together, or slow enough to be considered down for over a WEEK! Trying to contact customer service/log a ticket only gets me a generic automatic response. After 24+ hours, I get a reply, saying “Unfortunately, we’re not able to give you an ETA as to when the hardware move and restore process should complete” So, we’ll be down for a while, or not. Maybe intermittently. Being completely down for more than 24 hours is NOT intermittent! Not great, for someone who just launched her company on the local news. Fail. I’ll be moving as soon as everything’s up so I can actually access my data. Thank goodness my email’s hosted with Gmail! It could’ve been worse.

My web site is down too….but the ngix is running, don’t work only dinamic content…

Sent a support request last night, Apr 6, 2012, at 17:23:10pm, and still waiting for response over 21 hours later.

All my websites are down and cannot reach support. Where did the chat button go??? HELP!!!!

All my websites are down and cannot reach support. Where did the chat button go??? HELP!!!!

Another total downtime, besides own siter hosted at DH, even dreamhost.com – site itself cannot be reached.

restored miaplacidus PLEAAAASEEEE 7 days off, :(

I started today and I didn’t have answer about my account.
Please, I need only one plan.
I’m waiting a email with your answer.

same- all sites down, and i can’t login to the admin panel. feeling disappointed with the last year of service.

The support fail. I trying to find my “new error” here and now I come across several people complaining of slow … Bad job.

I’m down too…ugh.

Resolved??
I have the same problem. My sites have been down for a week, and now they are “Temporarily Unavailable”. Now it even fails when I connect via FTP or SSH. They made me read lots of wiki-how-to-tutorials to make me fix my sites myself, I wasted a lot of time and energy just to find that I’m not alone and that it’s (more than probably) DreamHost’s fault.
I just realized that my databases are gone too! I’m getting mad… and scared.

My site today has had intermittent but short bursts of downtime today. However, right now my site is down completely. Is anybody else having this problem? I’m on greengoblin.

All my sites are down, what is the problem?

My site has had all its files erased since sometime after Thursday afternoon. Your message keep saying that it is being repaired, but none of my other sites are affected.

No response to email support. No response to forum post. No number to call.

Is my site gone or am I just one of the few not to be restored? What did you do with my files?

Can I use backup?

WHY WON’T YOU GIVE ME AN ANSWER SO I CAN MOVE FORWARD???

So totally unprofessional.

please. my mysql server is down.

My Site is down. Is there a problem with the servers?

Looks like my site’s SQL server is down too.

I am having database errors too..

My site is back up now. SQL is responding, but we need to figure out why this happens. I frequently have users reporting SQL connection problems. I’m taking it up with support, but I wanted to leave this here in case any others are having these issues.

what happened again, websites and panel is nor reachable

Woke up today to work on some of my sites. But ALL of my 40 websites are not coming up. No Website. No Panel. I cannot access either through FTP or the web.

I’m on a dedicated server, and my site is still down.

I’m on a dedicated as well and my websites are still going through this outage.

VPS down, no MySQL…. not today. please NOOOOOooooooo!!!

Dear Friend,

I’m also on a dedicated server and my site (www.seguroviagem.srv.br) is still down…

Please help! Crucial applications running here…

Best regards

Same problem here. I have a dedicated server and all my sites are down.

All sites on our VPS are down… Please help!

Same here – server down……

All my sites are down today! Please update.

Sites were slow and flaky this morning, now all sites, FTP and MySQL are down

VPS and MySQL are both down. ps59632

MY SITES IS DOWN!!

All sites down on dedicated server. All FTP lost.

This is looking like another case of Monday. Dear Lord, please have mercy on my soul.

I have issues with two seperate managed servers – greengoblin – up , yedprior – down. I noticed that the panel has come up with a notice from dreamhost, looks like they are looking into it.

My dedicated server is completely down. GET IT TOGETHER!

This problem HASN’T been resolved; all my sites on my VPS are still down.

SORT IT OUT PLEASE.

Down again. Jizzz

Can’t you solve this? Wasn’t cloudfare suppose to prevent this from happenning?

I’ve got cloudfare activated and it didn’t stop my website to die!

Please sort it out

Hope Dreamhost can resolve problem faster….my website is down >_<

Regrettably, I am having a lot of prejudice!


Website Currently Unavailable
The website you are trying to access is currently unavailable. Please try again at a later time.

If you are the site owner, review the following resource to help resolve the issue.

© 2011 CloudFlare, Inc.”

I hope that in the next days this problems will be end , because here everydays for 3/4 hours ( business hours the site are down ) as now again . Thanks

argh my car park has crumbled… i mean the site is down…

sites down again .. they were up earlier however

ah yep a number of my VPS sites are down…

We just launched a campaign this morning and sent it out to 6000 people for a high profile brand. And it crashed just after we sent it out. Now we are getting slammed by people saying the site is down. Can we get an update please, DH?????

Site is not down, my websites have also been very slow and not working properly for the past week. All I got was automated email message support. Will be moving my sites, never again.

Ahhh stop stop stop , for all customer speaking about http://www.dreamhost.com is not professional is not REALLY , I have so many mame domains and so many to http://www.networksolutions.com sometime they also do not answer … for long time …if answer , because if do not write correctly english like at so many USA person also do not take care … They here from Dream Host , are professional only many times happen problem but they promptly take care the problem and fast …

VPS site is still down.

My site’s been having sporadic outages the past couple of days as well even with CloudFlare. What’s going on??

my site is still down this is horrible I thought dreamhost was supposed to be reliable this is my 5th downtime in 3months!
you say to email but you guys take 24hours to answer the emails!!!!

I just launched a client’s website, and convinced him to switch from 1and1. Now his site is down and I’ve got egg on my face for saying I thought Dreamhost was reliable.

hi there.. my two vps are down.

any helpppp

Fired! DH no longer deserves our business, or anyone else’s business for that matter.

give time

We need access to our files so we can move to another hosting

All my sites are down. What in the world is going on? This just happened two weeks ago. And three weeks before that. This is ridiculous. I’m on a VPS and am paying for better service and better hosting. What has happened to your service?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Improvements to outgoing spam prevention policy
4:02 pm

Improvements to outgoing spam prevention policy

Posted (April 4th, 2012 at 4:02 pm PST) by Brian@dreamhost

In the ongoing battle against spam it has become necessary that we prevent outgoing mail from spoofing other major email providers including: hotmail.com, live.com, gmail.com, yahoo.com, aol.com, earthlink.net.

Forging / spoofing email sender addresses to an address that a DreamHost mail server does not have a logical right to send as creates problems when dealing with spam and blacklists, particularly with major providers that are receiving spam from their own domains. This change is just one of many measures we are working on will help strengthen the reputation of DreamHost’s mail servers for all of our customers and help prevent future blacklistings.

If you are currently sending as one of the above domains via your DreamHost email user you will need to update your mail client settings to send as your proper domain email address. If you do wish to send as one of the prohibited domains (such as Gmail) you will need to configure that email account through the proper email server on your computer/device.

This may also affect those that have scripts that send email as a form-input email address, or a configured admin address. These forms/scripts/CMS settings need to be changed to send as a proper address at your domain.

If you’re affected by any of this, not to fear, we’re here to help, so just drop us a line here. We’re making these changes to address spam being sent from our network. If you’re sending legitimate mail, then we’ll make sure you’re still able to do that!

Update Apr 5th, 9:25am PDT: Just to clarify, we didn’t roll back this entire change, but just the aspects of it that were causing mail to incorrectly bounce. Customers still will not be able to send mail using outside domains through our system, including the ones above (hotmail.com, live.com, gmail.com, yahoo.com, aol.com, earthlink.net). Sorry for the confusion. -dan w

Update Apr 5th, 12:56am PDT: This change has now been rolled back due to major mail providers being blocked on our end due to the issue with the change. If you are still experiencing any issues, please contact support for assistance! -Patrick

Update Apr 5th, 12:20am PDT: This change is being rolled back due to major mail providers being blocked on our end due to the issue with the change. We will update this post again once the change is completely rolled back and services are working again. -Patrick

This entry was posted on Wednesday, April 4th, 2012 at 4:02 pm and is filed under DreamHost Features, Policy Changes, Security Updates, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

51 Responses to “Improvements to outgoing spam prevention policy”

It should be noted if at all you are using your own written code to change the From address to these addresses for ease of replying, you should instead set the headers somewhat like the following:

From: “Users Name”
Reply-To: users@email.tld

This will allow you to reply to it and see their name as from while still getting past the filters and being fine against spam.

From should indicate where the email is from, and reply to should indicate where replies should be sent.

wow. i’m trying to email one of your customers and my emails are getting bounced back since i use Gmail. Is this your intention???

I am having the same issues as Julie. This is ridiculous, how can I contact your customers?

yeah, can’t say this sounds like an entirely great way to solve this…

some forms are configured with addresses outside dreamhost so that replies can be handled with “ease”

for example, a lead generation form, will submit info to the form admin, and will have the return address of the user who submitted the form, so that when the form admin reads the submission in his/her email box, they can hit reply and it will then go the person who submitted the form. or, as many e-commerce packages will do, the order email which gets sent to a store admin, has the return address of the customer, (which might be gmail, hotmail, etc etc etc) so that the store admin can reply to the order, if need be, and it will go to the person who submitted the order.

so from what I hear, you are breaking that functionality.

another, not good from the old folks at dreamhost…but, another reason to continue my personal & business migration to another web hosting company…

I do hope you are not just planning on this being the only notification you are sending out regarding this…I think ALL of your customers should know about this…and not all of them look at dreamhost status.

anyway…what-ever…

I’m totally confounded. I can’t believe our hosting provider would pull the plug on our sitewide e-mail service like this without any prior notice or warning. Our entire organization is suddenly WITHOUT E-MAIL. We’re now dealing with dozens of e-mails bouncing back and all of this after a nearly 8-hour spell of downtime several days ago.

I am absolutely REVOLTED by the level of service I’ve experienced here in the past couple of months – and I’m not one of the typical over-reactors that comment on every status update.

You have completely pulled the rug out from under our tiny non-profit organization. Our hosting is free, and we’re STILL totally unsatisfied. Unacceptable.

So just to be very clear to Dreamhost support:

At around 8:35 tonight, you made some change that is now causing any email sent from a Gmail account with a gmail reply-to through Gmail SMTP servers to *bounce* upon hitting a Dreamhost mailbox.

This seems misconfigured. It’s also keeping Dreamhost email accounts from receiving proper mail.

We’ve tested it with multiple Gmail accounts and Dreamhost addresses. Nothing works.

Again, this is *not* email sent through a Dreamhost SMTP that has a Gmail reply-to — this is email sent from Gmail inbound to Dreamhost.

PLEASE FIX SOON

My emails are no longer working and I’m just using DH. No other ISP is involved. Nice job guys. Fix it soon or I’m moving my shared hosting to InMotion along side my VPS hosting

April 4th, 2012 at 9:01 pmKathi Derevan Says:

I am not able to receive any e-mail right now. My correspondents are getting a notice that my domain is rejecting e-mails. What??

April 4th, 2012 at 9:07 pmMr. Sosad Dreamhost Customer Says:

You know, personally what I believe is now going on is that Dreamhost is trying to devalue themselves as much as possible. I mean with the constant downtime, slow websites, HUGE security breach, and other unimaginable things that we don’t know about…what else could it be?

I’ve been with DH for a long long time now…but then again, some years back I was dated this one girl for a long time too, and she ended up developing this really smelly foot fungus…eventually I couldn’t take it any more and she and I had to part ways as the smell was just too much to bear.

My theater depends on the ability to receive mail. It’s how we get questions, get notices of orders, and so on. And this change seems to have had some unintended consequences.

4th time in recent weeks that I’m having serious troubles (as described by everyone else here). Seems crazy that anybody with a gmail isn’t able to email me… losing patience!!! Anyone home over there? I’ve sent support tickets and tweeted and haven’t gotten any replies and you aren’t updating this blog… what is going on?!?!

You have got to be kidding me. This is the lamest thing yet. WTF is wrong with you guys? Seriously! I’ve been forwarding my email to gmail for umpteen years. Been a customer at Dreamhost forever. Suddenly all emails to me are bouncing!

!!!!

I forward all of my DreamHost email to my Gmail. Since this afternoon I am no longer able to receive any email to DreamHost addresses.

I can’t believe that DreamHost would mess this up so badly. This is email we’re talking about. The most basic of internet services.

I’ve been a customer for something like 9 years… thinking maybe it’s time to pull the plug and just move everything to Google Apps.

Dreamhost needs to immediately post an explanation of what is going on and how it gets fixed. That explanation needs to be both more detailed – i.e. what are all the situations that might cause this problem – and more clear. As far as I can tell, you are changing something that has previously been explicitly supported by Dreamhost.

Perhaps more importantly, when you do something like this, you need to give people advance warning. Clients do not like their email being broken without notice – me included. Seriously, this is one of those times when I ask myself “What were they thinking?!?”

Finally, can you explain why the error message I get sending email to myself from an external (Gmail) account says “Google tried to deliver your message, but it was rejected by the recipient domain.” You appear to be rejecting incoming mail.

Actually, as I look at what’s happening, you need to back this out and start again. You need to do it fast.

P.S. I’ve been with DH for a long long time now…but then again, some years back I dated this one girl for a long time too, and she ended up developing this really smelly foot fungus…eventually I couldn’t take it any more and she and I had to part ways as the smell was just too much to bear.

Same issue here: Dreamhost rejects incoming e-mails that are forwarded from my Gmail account to my Dreamhost e-mail accounts.

Now i cannot send email from my Gmail account using Thunderbird.

Lamest fix ever.

Dont fix what aint broken !!

Mail to my dreamhost hosted email account is now essentially broken. Friends are getting bounce messages. I send test mail to my dreamhost account from my sbcglobal.net email account, and that’s not being delivered, either.

I hope this gets fixed quickly.

What is this, amateur hour?

I love how this change was just implemented…at the end of the day (night time here on the east coast)…

I have a bunch of customers on DH (using separate accounts too), and most are forwarded to gmail (a suggestion given to me by a DH support person some 5-6years ago)…

This number of accounts which I’ll have to fix probably numbers into at least 50-75.

all is broken…well, once again…just when you think they couldn’t surprise you any longer, Dreamhost has decided to take their already failing service and make it worse…

Trevor – how do you even fix it????

you know Seth…right now that’s really really good question…

in the meantime…emails are straight failing…

some are getting bounces…others I’ve tested don’t even generate a bounce message…and of course, the message doesn’t arrive as it should…

no warning, no suggestion of how to fix…and as expected…no response from DH either.

This is Dog, once again. I’m having the same issue as Trevor. At least before email were bouncing, now they just disappear into the nexus that is the shithole of dreamhost. Off all the shit these rocket scientists could tinker with, they pick this.

FWIW it might be better that they are not bouncing– hopefully that means that they will come through at a later point rather than getting bounced outright (at least this way I’ll eventually get emails people are sending me… I hope…)

for me…my test is sending to myemail@mydreamhostdomain.com which forwards to my gmail account…when I sent from yahoo…no bounce, no error, and not received…when I sent from my hotmail account, at least I got a bounce error…

I am just sitting here in total disbelief.

I seem to have suddenly stopped *receiving* email (from any source I have tried)….

Fucking bullshit is what this is. So you are punishing everybody for a few’s sins?

I am so sick of Dreamhost.

Anybody use Opera? The mail program has this nice redirect function where it will forward an email to another address without all the forwarding nonsense like the little >’s. I use it to help me manage my many email addresses if someone sends something to the wrong email address. Now, I can’t use that nice function.

YOU SUCK!

How about just identifying the actual spammers and banning them from your service, instead of punishing us all. Oh wait, that would be actual work, sorry about that.

Yep. Nothing working on my end. Even emails from dreamhost to dreamhost. Nothing. Every time DH does something I say to myself I’ll switch. Then I never do. My own fault. But this time I’m going. Moving far away from all of this.

things seem to be working again…so that’s good…hopefully it will stick, and DH will fix the problem long enough for me to move websites and emails onto another webhost…

FYI, all of my email was down but within minutes of submitting a support ticket it’s back up for me. I’d say if anyone is still down, contact support.

oh, and I must say I love the last line in this notice about “If you’re affected by any of this, not to fear, we’re here to help, so just drop us a line here.” yeah…right…as if…

I’ve tested a contact us form on my website, as it uses php mail() to send using the submitters email address.

It worked when I sent using a gmail account. Anyone else able to receive on their webforms where it submits as sender = gmail account ?

@Trevor Ryynolds

I cant knock dreamhost support, I think they are first class (the guys, not saying the equipment is). While we may never get that reply as quick as we all want it, the replies for months have been specific to my account and query, included explanations, solutions and further reading. Seriously, I dont get half the response from other providers in any industry.

Do give them a go and as “Jason” above (great name) proves, his was quickly resolved. It seems like all you have to do is raise your hand and prove what you want to be whitelisted is harmless and they’ll fix you up.

Anyway, I’ve submitted a ticket to confirm if the contact us tests I did were to be expected or not. I’ll update here any response, unless someone beats me to it.

I wanted to say that fight against spam is very important. I guess it’s hard, but it’s important.

@Jay, by the time you had chimed in on this (and the time that @Jason commented here), Dreamhost had already rolled back the changes…there was at least about 4-5 hours of failed email delivery. Some years back, I took the same stance as you regarding DH, and their wonderful support staff (even with somewhat slow replies). They WERE the best; but here recently, major security breaches, multiple hours of downtime, and degraded performance (of which DH *always* claims it’s something with your website (wordpress, drupal, etc). I had to look around, but I did find another webhost with unbelievable support (always there, always quick to respond – and a voice on the other end of the phone if needed).

@Julien, the fight against spam is important, for sure…but there are much better ways of dealing with it than deploying blanket rules which will adversely ALL email on a network. This includes dealing with spammers one by one, and using tools such as blacklists and domainkeys (among others).

and deploying changes which WILL adversely affect email delivery (or any service for that matter), without any warning to customers, and then; as I see it, walking away from it (I know how it goes, I played the engineering game for a long time) is 100% unacceptable service. At least in telecom, if someone did something like this; they wouldn’t even be kept around long enough to fix it…

@Ash, moving sites (especially if you manage a lot of them) is a time consuming process; which has kept me in place here at DH much longer than I care to be. But after their last bout of about a half day’s downtime (Mar 23rd), I already started with a plan to move sites to another host.
best wishes to all…

I have been using dreamhost to push e-mail under my gmail address for years and never had a spam complaint. Why can’t users with an established e-mail history of NEVER spamming through dreamhost be grandfathered in some kind of whitelist? Considering the unannounced, massive loss of functionality to many users, finding a way to transition the limitation in (or forgo it entirely) for long-term customers is really the only professional route. I’m sorry if DH has difficulty protecting their servers from spammers, but don’t punish me for it!

While this is a good idea in theory (and practice), this is exactly indicative of my frequent frustration with DH update policies: A significant operating policy change should be predicated with plenty of warning. I appreciate that they are open and descriptive of the change, but forewarning should be standard practice. Basically, “Today you might notice that you can’t send email” is not sufficient. Then it ends up offending some “major mail providers” (read, “more important customers”), so they roll back. I really do like DH, but…

DH…PLEASE provide adequate notice so that scripts can be changed and tested. On the customers end and on YOUR end.

This entire thing could have been avoided if you gave a one month window for testing and switching. Please start to realize that when you make changes like this you AFFECT customers’ business. Please, start to provide professional courtesy to your customers. It will go a long way. Who ever decided to flip the switch on this without any type of warning to customers so they could change and test their scripts should be fired.

With DMARC: spoofing (ie. what you do with your forms) the “From: ” header will become useless; better you learn this way.

None of our email accounts skipped a beat.

I do agree with Janice though, notice would be nice.

Still having an issue… despite making the changes needed… and STILL waiting to hear from customer support.

Thank you Dreamhost for shutting me down last night and today. I really appreciate. (sarcasm here)

again my websites down… very bad service! change now

April 5th, 2012 at 11:26 amDaniel James Says:

This is ridiculous. I’ve been a customer for ~twelve years and you are really screwing the pooch these days. Rolling out this kind of change without warning and with catastrophic faults (bouncing all email from gmail? You jokers) is unacceptable. Time to move on, I suppose.

If anyone is having trouble with php scripts using php mailer, etc. I signed up with SendGrid to use their SMTP service. Got the majority of my scripts running flawlessly now.

Still have not heard the word boo from Dreamhost support on a support ticked I submitted last night and today.

Thanks again Dreamhost, The extra work last night and this morning, has been a true joy. It really has. (Yes, this is sarcasm).

I find it completely befuddling how little you care about the impact this has on your customers. There is 0 excuse for not giving some type of warning on this. This was not an emergency patch, etc. You could have warned us, and you chose not to.

@Trevor Reynolds: Ah that may explain why my tests seemed to work despite the advice above. No reply yet on my ticket, but then I didnt put a high priority on it either, as things seem to be working. Lets see what update we get from the DH team.

Still waiting a response on my high priority submitted ticket placed at: Apr 4, 2012 – 19:07:51 / #XXXXXXX

Is anyone even there?

Quoted from your post:
“If you’re affected by any of this, not to fear, we’re here to help, so just drop us a line here. We’re making these changes to address spam being sent from our network. If you’re sending legitimate mail, then we’ll make sure you’re still able to do that!”

At this point I have everything to fear. I am shut down on two sites, made the suggested changes (sending from the domain name) and have gotten zero response. W/O DH’s help and extra expense to me, I was able to use an outside SMTP service to bring other sites online. But I am still waiting for a a response from DH on 2 of my sites.

@Janice -agreed on the impact. It is amazing how little Dreamhost seems to respond to any mail issue. And I can also vouch for SendGrid. Switched over to it in February when Dreamhost couldn’t keep themselves off the blacklists. I still get some bounces, but the diagnostics are good (and I was able to link directly into a Dreamhost PHP page to record the bounces in our database). Outgoing connection to SendGrid was painless from my PHP scripts too. [I am NOT a shill – just a working stiff who needed a solution that worked too!]

I get that you’re trying to be good netizens by policing outgoing mail for forged addresses, but you can add me to the list of those surprised and annoyed by this change.

What I don’t get is: Shouldn’t requiring SMTP AUTH to the mail server already prove that we are who we say we are, regardless of what email address we use?

Not being able to obscure my email address with a throwaway remailer (like mailinator, or Craigslist temporary addresses) is going to get me *more* spam, even if it does cut down on outgoing forgeries. This is even more of a hassle than having to install a custom CA cert for SSL IMAP and SMTP. Both these policies end up making it harder for Dreamhost users to keep their email addresses and passwords secure.

I am trying to send email using php mail()-function but it doesn’t send email to any address. Why isn’t this working? It worked before!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Planned maintenance on ‘ash’ web server Tonight Monday, April 2nd @10pm (PDT) (RESOLVED)
5:25 pm

Planned maintenance on ‘ash’ web server Tonight Monday, April 2nd @10pm (PDT) (RESOLVED)

Posted (April 2nd, 2012 at 5:25 pm PST) by Dan J

Tonight Monday, April 2nd @ 10pm (PDT), the ‘ash’ servers will be moved to a new chassis. During this time, any sites hosted on this machine will be unavailable. This will not affect email or MySQL services, only web services for domains hosted on this server. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

We expect the maintenance window to last less than 15 minutes and we will be updating this status post with information as the upgrades progress. We’re sorry for any inconvenience this causes you.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you.

Update Apr 2, 10:24pm PST: The move has now completed. Everything went smoothly, total downtime about 10min. If you have any more trouble then please contact support through the Dreamhost panel. -Dan J

Update Apr 2, 10:00pm PST: The move is about to start. We will continue to post here with updates. -Dan J

This entry was posted on Monday, April 2nd, 2012 at 5:25 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

33 Responses to “Planned maintenance on ‘ash’ web server Tonight Monday, April 2nd @10pm (PDT) (RESOLVED)”

I SUBMITTED A SUPPORT TICKET ABOUT MY SITE OVER 5 DAYS AGO AND HAVEN’T GOTTEN A RESPONSE. I’M ON STAMPEDERS AND MY SITE HAS BEEN EXTREMELY SLOW AND/OR DOWN ALL YEAR. DREAMHOST GIVES ME A DIFFERENT REASON WY EVERY TIME. CAN SOMEONE RESPOND TO SUPPORT TICKETS PLEASE? I’VE BEEN A CUSTOMER FOR MANY MANY YEARS BUT AM FED UP AND WILL BE CLOSING MY ACCOUNT IF THE ISSUES WERE RESOLVED.

WHAT HAPPENED TO DREAMHOST, YOU USED TO HAVE GREAT AND QUICK CUSTOMER SERVICE. NOW YOU DON’T RESPOND AND WHEN YOU DO IT’S NOT HELPFUL AND IT DOESN’T FIX THE ISSUES!

I have 3 support tickets out and NO response. Our site resides on ras-alhague and it is practically unsuable with 40 page load times!!!
We are desperate. What is going on Dreamhost?

Cannot connect for two days. Re set my password. And still get
The same reply. I.e. wrong entry. I need help this is my business!

my site is down :(

I’m on miaplacidus, all my sites went down since sunday, nothing has changed since then!
all i heard from support is that a hard drive crashed and that they are still restoring data from backup!!
I’ve lost a lot of money because of this unprecedented 4 days outage!
Dreamhost is turning my life into a nightmare!

my webs are down for 5 days and still not working! i dont know what te do

my sites are on miaplacidus too

My site(s) are on miaplacidus… Clients are getting upset and I still haven’t received an ETA on when it would be fixed.

What is going on with your mysql servers!? Why haven’t you posted any information here!? It’s been over 4 hours now. All of my clients are being effected.

Database error error…. again. Rebooting failed.

Dreamhost reliability has seriously gone downhill in the last six months. I’m now researching viable alternatives. Pity really…

April 3rd, 2012 at 3:03 pmJonathan Tonge Says:

I can’t connect right now. Internet is fine. I just can’t log in to FTP with Dreamhost. This has become a weekly thing.

Shame there’s not a single line for REVOLUTION, which had a catastrophic disk failure and is down for over 3 days now!

On panel it says “your happy engineers are working”. Maybe they shouldn’t be smiling so hard and concentrate on getting it up again.

As stated by others, dreamhost stability is going downhill.

April 3rd, 2012 at 3:29 pmJonathan Tonge Says:

I wish my DreamHost account would expire soon. I just did a review on one of those web geek sites explaining how bad DreamHost has become. In your last outage, 1,500 people cried out about how bad things are. And they weren’t angry just about the current outage. It’s weekly, semi-weekly. You need to start firing some people and send the right message – this is not ok. I’m not talking about your support staff, but the managers who are in charge of maintaining and upgrading the servers, and the executives and accountants who approve the budget. Your marketing team better be furious about this, and if they are not, you have a good reason to fire them.

PLEASE RESPOND TO OUR SUPPORT TICKETS!!!!!

We should not have to post complains here to get a response!!!

No access via FTP on ksora…sites are still up though but can’t do what I need too.
What are the alternatives to Dreamhost? By alternatives I mean price wise vs alloted space and ability to host multi sites.

I wish I’d read this before signing up yesterday. I just set up my site and it’s been down 3 times in the last 24 hours. No response to tickets. No status updates reported. Disastrous customer service.

My email is unable to be sent again…. websites seem to be up though. Email is crucial, people. This is the last straw for me. Time to move!

This is terrible. My site is down again. And, yes, I have submitted a ticket. I am sure dreamhost will blame it on some sort of denial of service attack. Oh well.

This comment string says it all. DreamHost is failing! No replies to customers who are losing money over the service. Just change providers.

ALL MY SITES DOWN HELP

My site spooool.com is hosted on Mogadishu and is offline, this is getting a bit too regular for my liking… Any response guys???

helpppppp site downnn

My site is back up now. Think it was down for at least 90 minutes…

April 4th, 2012 at 10:57 amJonathan Tonge Says:

FTP was down between 4pm yesterday until this morning, then it was back up this morning, and now both front-end and back-end don’t work. Internet is fine. This is all you. Was it just an accident that I was back up? Should I report to you when something is working so you can break it. This is disgusting. I’m spending dozens of hours this month working around you lousy service. The only thing preventing me from switching is the time involved, but in hindsight it would have been a great investment to switch to someone else.

This is getting personal.

April 4th, 2012 at 11:15 amJonathan Tonge Says:

The problem with server reviews, and this especially pertains to DreamHost, is that they pay people commission to write good reviews. Well actually they pay people to refer them to DreamHost through their affiliate program. But of course that implies selling good reviews.

Hostgator and inmotionhosting look reliable. inmotion claims 99.9% uptime as measured by an independent third-party. That sure beats the 90% uptime I’m experiencing with DreamHost this year. It’s just pathetic.

April 4th, 2012 at 11:30 amJonathan Tonge Says:

How can we go about getting our money back. You’re not holding up your end the of the agreement.

So right before submitting my ticket, my site and FTP go live. Been about 15 minutes and things are well. Now site is down again and FTP is ok.

How is this remotely acceptable. What are you doing with the jabber.

WTF

Spoke to soon. FTP just went down. Phew. For a second there I thought something was working and was like “strange”. I certainly don’t want anyone to see my site or be able to add to it. Great work guys for making sure nothing is working.

I want off my server now.

portvila themountains brock war jamestiberius

It obviously is broken. Put some other poor customers on it that you don’t care about on that one – It will for sure make them flee.

Still out. Going to write a few reviews on review sites while I wait for you to take care of this.

5:39 ET FTP is back up. Who has a stop watch to see if the FTP goes down before the front-end site goes live?

5:44 ET, HTTP is back up.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues with one of our datacenters ( RESOLVED )
11:49 am

Connectivity issues with one of our datacenters ( RESOLVED )

Posted (March 23rd, 2012 at 11:49 am PST) by Brandon

We are currently experiencing some connectivity issues to one of our datacenters(more specifically, there is a small subset of our network which is unresponsive, fortunately the majority of our network and customer base is not impacted). Our networking, system engineers and datacenter operations teams have been deployed and are investigating. Sorry about the inconvenience this is causing. Please check back here often, as we will be posting updates as we receive them.

This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.  Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. As of 12:50 pm we are currently in the process of re-routing traffic around the affected networks in order to restore communication. We don’t have a concrete time to completion just yet, but we are not anticipating much longer before there is noticeable improvement. We will continue to update this post with new information as it becomes available.

Update Mar 23rd, 5:01pm PST: We are happy to report that all services were restored at 3:30PM. We have been spot-checking various services to definitively confirm that the network was fully functional. Our network engineers will continue to actively monitor the stability of our network. We apologize for any inconvenience this has caused and appreciate your continued patience. If you happen to notice any problems, please contact support immediately. -Brian S

Update Mar 23rd, 4:26pm PST: A good portion of network services are back online. E-mail is now being delivered, but again there will be a delay of 4-6 hours for all e-mail affected by this issue. We’re still working on the odds and ends, and will provide further information here when it becomes available. -Brandon

Update Mar 23rd, 3:50pm PST: Services are continuing to come back up. All e-mail affected by this issue will be sent, however may be delayed for 4-6 hours. We’re still monitoring the situation and will provide more information when it becomes available. -Brandon

 

Update Mar 23rd, 3:30pm PST: Services are coming back online at this time. We’re continuing to monitor the situation, and we’ll update you again as we get closer to considering this issue resolved. -Brandon

 

Update Mar 23rd, 3:10pm PST: We’re still working on restoring services as quickly as possible. Unfortunately we’re still experiencing network instability for which we apologize. We are aware that this is causing panel service interruption in addition to impacting other customer services, including web sites and e-mail. Our networking, system engineering and datacenter operations teams are working feverishly to resolve these issues. -Brandon

 

Update Mar 23rd, 2:20pm PST: The uplink which was causing the initial problem came back online; however, when we attempted to integrate it again we experienced network instability which necessitated a reboot of our core routers. That is still in process after which we will be working on stabilizing the network and ensuring everything comes back up and is functioning normally again. -Brandon

 

Update Mar 23rd, 2:00pm PST: Upon further investigation we’ve found that one of our uplink providers is experiencing an outage that’s affecting these networks. We have redundant routes setup to prevent this type of effect when a provider issue such as this one occurs, however the routing tables in question did not self correct, and we’re now in the process of correct that, which should take care of the issue. -Brandon

 

Update Mar 23rd, 1:40pm PST: We’re still working on resuming communications on these internal networks by routing traffic around the problematic machines. This is taking a bit longer than previously anticipated, but we’re still working diligently toward a resolution.  -Brandon

 

 

Update Mar 23rd, 1:18pm PST: Traffic is now being routed around the problematic section of our internal network, and things are beginning to improve. We’re still working diligently toward a full resolution, and will update this post when we have new information. -Brandon

 

 

Update Mar 23rd, 12:50pm PST: We are currently in the process of re-routing traffic around the affected networks in order to restore communication. We don’t have a concrete time to completion just yet, but we are not anticipating much longer before there is noticeable improvement. We will provide additional details as soon as they are available. -Brandon

 

 

Update Mar 23rd, 12:40pm PST: This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.  Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. Our networking, system engineering and datacenter operations teams are still hard at work, and we will update this post with further information as it becomes available. -Brandon

 

Update Mar 23rd, 12:20pm PST: This should only be affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.  Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. We’re still investigating the issue, and will update this post with further information as it becomes available. -Brandon


Update Mar 23rd, 12:06pm PST: This should only be affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.  We’re still investigating the issue, and will update this post with further information as it becomes available. -Brandon


This entry was posted on Friday, March 23rd, 2012 at 11:49 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

1,479 Responses to “Connectivity issues with one of our datacenters ( RESOLVED )”

Webmail is down for me.

Oh nooooooooooooo, not again. Please move the entire Homie cluster to a different hardware in different location!!!!!!!!!!

March 23rd, 2012 at 11:54 amJaime Linhares Says:

:S

webmail is down for me. can’t even ping my server :(

Site is down. showing database error. Trying a reboot. Pop email was acting up earlier.

Still having issues with webmail.

Our website appears to be affected.

we can’t connect to homiemail-sub5 , pop and webmail!!

reboot of mysql server failed.

MySQL server is dead.

Gaaaawwwwleeeeee….

Yeah, my webmail is inoperative too. Web site itself is okay.

sub5 mail server is down!

Does anyone know how is hosting WordPress on Amazon?

lol, imagine that? pfft.

This is the third time we’ve had major downtime with you guys. No more, we’re leaving.

I need you to fix this problem soon and neither are many flaws in time …..

Amberjack (MySQL) is down, and I’m not sure about Mimosa…

I hope DH can get us back up quickly this time. Your service used to be very reliable, but this year- not so much. Please get it together DH.

I can not access my webmail or send mail from my clients

We pay the prices we pay and still substandard service!

Unbelievable.

Getting frustrating that this seems to happen every few months. Every time I try to build a user base the site goes down on a busy day and my traffic slacks way off. I really don’t want to go through the hassle of changing hosts and making back ups and all that, but you’re all making it very difficult. Where’s the redundancy?!?

I keep thinking that you’ve had so much downtime, you can’t possibly have any more. But then something goes down again… I should seek help!

cant get or send email and webmail is no use. ….. in the middle of a job. Thanks dreamhost… this happens a little too often for my liking!

I feel like an idiot for staying with DreamHost. It has become a reliable assumption that I will have problems.

Good to hear it’s not only me…
Seriously. Get us back alive!!!

I second this…

“Please move the entire Homie cluster to a different hardware in different location!!!!!!!!!!”

This is literally a weekly experience. Not like “I just say weekly to act dramatic and angry”, but literally weekly.

March 23rd, 2012 at 12:02 pmWilliam Teach Says:

Apparently, they forgot to fed the hamsters that power the servers again.

if it were up to me, our company would have dumped dreamhost long ago. unfortunately, it isn’t up to me. still, we run 1/2 of our business through email, and when it’s down, we’re losing thousands of dollars of productivity. you need to fix your stuff, this is getting ridiculous.

I’m wondering if this is why I can’t update my svn projects right now, was able to access them a few hours ago, but no access now
“Error 118 (net::ERR_CONNECTION_TIMED_OUT): The operation timed out”

5th time in a few months. You guys should be really, really good at fixing it by now.

Can you provide more info and maybe an estimate of the duration of downtime?

Oh my GOD… nOt again!!! pleaseeeee dO sOmthing tO stOp all this prOblems …. all my emails are dOwn, hOpe they get fix sOOn, I was sO happy with yOur service, but this year is getting really bad…. please get it tOgether!!!

Ugh… site won’t load, can’t access webmail, nothing.

I’ve always supported you guys in the comments here when everyone complains, but downtime has been happening so frequently lately and we’re losing our client’s faith in us. You seriously need to take that 100% up time guarantee off of your site. It’s not even an exaggeration anymore, it’s just a lie to potential customers.

homiemail-sub5 is down. including webmail. been a customer since 2001 but i think i’ve had enough.

DH people-
Please get this chronic problem fixed. You came highly recommended, but my email has been down 3x in 3 months. Had I known this beforehand. As it is, how can I recommend DH???

I guess this explains all the database errors.

Web mail down here, server is homiemail-sub5

Have filed a trouble ticket. This has been a slowly developing issue since our outgoing email has been rather lame for a couple of months. I also note that the webmail portal isn’t taking imap connections anymore either.

Agreed @brian REDUNDANCY!!!???

March 23rd, 2012 at 12:05 pmJaime Linhares Says:

Your server is a big shit! I need to send emails!

March 23rd, 2012 at 12:06 pmJaime Linhares Says:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Some of me emails and seemingly all of my databases (shared and VPS) are down

March 23rd, 2012 at 12:07 pmJaime Linhares Says:

is the fourth time that the server down OMG

Second day the website has been down. Yesterday it was an internal error and today it is:

General Error
SQL ERROR [ mysql4 ]

Can’t connect to MySQL server on ‘mysql.forum.XXXXXXXXXXXXXXXX.com’ (4) [2003]

An sql error occurred while fetching this page. Please contact an administrator if this problem persists.

Glad I am not alone. I am looking for a new server to host and move away from DH for this unreliable service issues regularly. And we have been asked to pay more suggesting it was our fault for using too many resources. Like hell it is. Just excuses to get more money!

How can we say this is 99.9% up time people! I have 12 websites down and one of those is a major one I am trying to promote!

Regards,
Jimmy

dreamhost can suck a dick. down like every month now.

anyone got any reco’s on a GOOD provider?

AWESOME! NOT! We need this fixed ASAP. We just go over a hacking issue and now this? Thank God my mail is google hosted!

This is really starting to get ridiculous. You’ve been in this business for years; yet the frequency of downtime makes you seem like rank amateurs, and every subsequent excuse feels more hollow.

It time to jump ship. With this rate there will no money left to host after losing thousand of visitors.

Site is down due to a database error.

For the price that we pay, there is no reason to accept this much unplanned downtime. We’re leaving dreamhost at the end of the month.

Right before a major demo of an application I wrote. Worst timing ever. Also Changing hosting due to the ridiculously chronic downtime. I want my money back.

DH server goes down more than a paid escort in Vegas

Seriously does anyone know of a good hosting service they know of? Post it on here!

Regards,

Jimmy Deguara

Please fix this problem and let’s hope this is the last time this happens..

Rock solid guarantee indeed!

my website is in the process of a redesign and guess what when re-launched wont be using dreamhost!

DH you used to be AWESOME!!!!!!!!!!!!!!!!!!!!

WHAT THE F*#$ HAPPENED?????????????????????????????????????????????

Guys…I love ya. I’ve been with you for over 6 years. But you got to fix this downtime problem I’ve been seeing. I understand that shit happens, but fallback redundancy doesn’t seem like too much to ask. My mail isn’t working, wordpress stuff is really spotty…please move the homie group.

This is really getting ridiculous. This constant downtime is really affecting our organization and our ability to get things out in a timely fashion. Dreamhost needs to consider comping all of us for lost time, giving us a period of free hosting, or just getting their shit together.

you got 57 messages in 15 minutes, are we just a small subset?

This SUCKS!!! I can’t believe I’m this dependent on email but that’s the world we live in….I can’t finish off the business I was working on…I can’t run a business this way. I guess all Hosts have similar problems.

No email, no webmail….website is up and functioning…

Rather than send us an apology how about a credit for the business lost?

If the true American service came into effect ie money back when service is down, we would not have to pay anything!

Jimmy

You really should change your company name from DreamHost to NightmareHost.

This is the 4th time since January “connectivity to your datacenter” has taken down one of the $150 / month VPS accounts I help manage.

Wow.

DreamHost. I’m done with the downtime. Moving to a new host at the end of my billing cycle.

I think Sam put it very well: “I feel like an idiot for staying with DreamHost.” Me, too.

The definition of insanity, so they say, is doing the same thing over and over and still hoping to get a different result.

Staying with Dreamhost. There’s a new definition of insanity.

Please fix this! Too many email outages recently. Email is down. Homiemail-sub5.

Here we do again, so far I know I have a mail server down, now I have to go check sites.

Someone said “anyone got any reco’s on a GOOD provider?”

Well I use more than one hosting company and one other I can recommend is http://bit.ly/Hostga if you use the code “remember25″ you get 25% off.

What was interesting was that when 7 of my sites got hacked on DH All I got was a report, but on http://bit.ly/Hostga I had 15 sites hit, but they were able to just fix it and my sites were back up within an hour of their scan.

Why can’t DH do this?

i can’t believe i am asking this. have been with DH for almost a decade.
but anyone got recommendation for web host i can migrate to?
DH have been a very bad disappointment lately.
this is the final straw.

We can not connect to SMTP.

Please check! Our faxes, releases and other services depend on it.

Thank you,

Well, now our website is down too. We are on server: cepheus and our mysql database is on megamouth.

Warning: mysql_connect() [function.mysql-connect]: Can’t connect to MySQL server on ‘mysql.allcrewagency.com’ (4) in /home/allcrewdh/allcrewagency.com/home/includes/adodb/drivers/adodb-mysql.inc.php on line 355

Fatal error: Call to a member function CacheGetAll() on a non-object in /home/allcrewdh/allcrewagency.com/home/includes/classes.php5 on line 303

am filing another trouble ticket.

anyone like Fatcow?

Again!
DH, you just lost another customer…

really? again?

man.. i really really like(d) you guys. and really really want to like you guys. but, it really is getting to the point where you guys are losing your sh!t.

i say, spend money, do what you have to do.. otherwise, a year from now.. you are tanked.

One more to the list of people with issues. I’m seriously not happy and have been in the process of moving things for a few months now, ever since the SMTP server got spam blocked, but at least I could read inbound. This is annoying,

i’m migrating to webhostinghub. we shall see.

btw, i should note.. i have been with you guys for 5 years.

Holy god… this is a joke.

Again? it seems like too many probiems in mysql in the last days

have been with DH for almost a decade. Me too! I used to like DH and then they tried for a year pushing me to a server which is pay based on resources which is just an excuse to get more money. I paid more money for crapping service.

Anyone can post another good hosting service here. Please!

Jimmy

This is starting to get ridiculous. We either need an explanation of why this keeps happening and a proposed solution or some sort of refund / acknowledgment that this is not acceptable as a regular occurrence. If this is business as usual… I’m out.

THIS IS SO FRIGGIN RIDICULOUS!!! My Site completely DISAPPEARED from you servers for a few days about a month ago and now this!? I am losing valuable time here, DH! PULL YOUR CRAP TOGETHER!!!!

The funny bit is that I was gonna pay my yearly re-up today. True story! I’m not sure if I am now…

“Connectivity issues with one of our datacenters.” Why can’t you then fail over to another datacenter?! Where is the REDUNDANCY in this system?!? I’m getting so tired of these outages! Good thing I’m just hosting hobby stuff, if I had clients on DH I’d be FrEaKiNg OuT.

This is insane, dh whats going on, if you can’t get more dependable in the few hours, I will have to move all my sights to a different server, I need my e-mail to make a living, I can’t afford this sort of downtime.

i have been a huge fan of dreamhost over the years, specifically because they had their tech shit together better than anyone else at thet ime. but i have to agree with others here that recent outages have been frequent and absolutely unacceptable. my client, who hosts MANY websites here, is getting fed up, and i am ready to finally suggest he changing hosting companies. it breaks my heart, because dreamhost had the most tech-savvy support i’ve ever worked with. but when there is downtime like we’ve seen for a number of months now, it goes beyond what can be considered acceptable growing pains. dreamhost seems incapable of supporting their customers at this time. that is going to get spread around the internet.

I agree with several other people here, whatever is causing sub5 to REPEATEDLY fail, please change it. This can’t be just random chance, we’ve been down over and over. Simply put this is bullshit, and I am not alone in saying I want it fixed. I don’t want to threaten to leave, because let’s face it, that is a pain in my ass. You all count on it, so that I’ll stay, but I’m telling you now, I don’t want to leave, I want you to do your job and keep my services online so that I can do my job.

Me too!! This is the 1st time in 5 years of being with DH …. This is becoming a too often occurrence …. get it fixed or Im gone.

Me too!! This is the 1st time in 5 years of being with DH that I have posted. This is becoming a too often occurrence …. get it fixed or Im gone.

I’m down once again… Way to go!! woot Woot!!

I guess with every dream, you have to have a dose of reality! d’oh!

Not cool.

Love it. So glad we have all 5 of our properties on the same Multi-site WordPress set-up. We came because someone recommended it. Now, they make our business which services 5000 people at physical events seem unprofessional. It’s real money being lost by being with them instead. We are out of here as soon as we can.

I agree with Jake. Please fix homemail-sub5. Buy a new machine.

OK, so I have been with DH a few years, and I get that the rates are rather on the low end. I read lots of threats of leaving. I get that. I’ve stuck around because there is a little of ‘you get what you pay for’ in play here. However, when I can remember, in all-too-recent memory, a number of outages, that’s when the rubber (silicon) is going to hit the road. My renewal is this year. At this rate, it’s unlikely DH will be my provider when decision time comes, and I am making inquiries for alternatives now. Get it up DH. Ciao.

I pay for a dedicated hosting for this ????!! Down 4 times in 2 months !!

RE: Webmail, mail & MySQL

Add FTP to that list. Nothing on my site is responding and I can’t update my site either.

After the last outage I started moving my clients away from DH. Right now I can enjoy my lunch and relax rather than having to explain yet another outage to my clients. We still use DH for development and backup. I sleep alot better now too.

is it hard to move everything over to another ISP? I am scared to do it

This is unacceptable. Have we lost interest in the hosting business or something? How can this possibly keep happening so frequently? Get a different datacenter if this one has such terrible connectivity issues.

Please fix the damn problem already.

My site is down too. :( UGH.

No consolation if you are part of the small subset

i mean host.. not isp. but whatever. im not getting paid to know things

my private server is down, and since my private server has my MySQL and Memcached server, all my other shared hosting sites are down as well!

Shocking you &$%&^$ are down again!
These guys hired IT folks who must have learned their trade from cracker jack boxes … i’ve never ever ever heard of an ISP SUCKING so badly at, well, beingan ISP.

SHAME SHAME SHAME ON YOU HELLHOST!!!!

why can’t they keep our site up like how they keep this status site up!?

or do they host this with someone else? i will love to know who.

It’s the same crap every time.. “Connectivity to one of our datacenters”.. “small subset of customers.”

Well guess what… THIS customer who pays $150 a month appears to be affected every time.

So if it’s such a small subset of your customers.. why are you still doing business with this datacenter that you only use for a small subset of your customers?

Actually, why am I asking you a question? You don’t respond, and are absolutely full of shit anyways. Small subset my ass.

In case no one has mentioned, comments here aren’t the official way to register your dissatisfaction. They may or may not pay close attention to them. Write up a support ticket… THOSE they count.

I’m out. Although the DM support is very responsive and helpful, I cannot get past the frequent downtimes.

I’ve been with DH for over 6 years and I’m getting a little^Wlot tired of it too, especially now that I want to start having real visitors see my site, not just hobby stuff. One more chance.

Dreamhost?

More like NightmareHost.

Eric: Yeah right.. I wrote up a support ticket last time we had one of these widespread downtimes and I got a token form e-mail response and my ticket was canceled.

why do you think they call it “homie”? because homie don’t play that.

i left DH for a while, been a customer since 1999. i come back and two days later this happens?? nice.

Happening much too often

Moving to Amazon asap. Can’t run a business like this.

Im my case (tygra) domain is very slow to get a simple static file, no MYSQL, mail or webamil in this case just a small plain text file is taking almost a minute to load :( Pleae FIX

COME ON!! I can’t send email…..

Ticket also filed. All my sites are down because the mysql server is down!!!

Well, I am one of the customers affected by this issue, and i’m not happy with that. Not happy at all, I must say.

I AM LOSING MONEY!

March 23rd, 2012 at 12:27 pmJordan Fisher Says:

FTP and MySQL down. Again?

Please get this stuff up and running. I was supposed to get a lot done today and now I can’t.

I want compensation for damages!!!!

My site is down, not just mail! How long will sites be down??

i am getting really pissed with DH. all my website is down! shared, VPS and even dedicated!!! wts

BTW, you are keeping the world from hearing my awesome podcast. I hope you are happy.

This is ridiculous! MySQL is not working! Quite nightmarish for a site that runs on MySQL (WordPress)!

This is outright unacceptable.

Dream Host, pull your D@@# Hosting together!

why am i always a part of a “small” group of people that has issues. Seems that things that effect everyone are called small as there is no way i’m effected every time if its always only a handful of customers. at least be honest here.

This is so much fun, I just spoke with one of my clients telling them that now everything was fine and running smooth and then this… lol

Now I will sound like a sleazy car salesman.

Define “small subset of users”

I always seem to be within this subset. How does one go about getting themselves off of Homie?

Mysql server is down

Anon: Writing a support ticket may not get satisfaction directly, but the more they have the more likely they are to take action. They might not take action anyway, but if too few people write up tickets it’s even more likely that they won’t.

Or, you can just vote with your feet, but they might not know why.

This is getting REAL old REAL fast. Our clients are not happy with us again. You may be causing us to lose business soon and that means you will then lose our business. Please get this fixed ASAP.
Did you grow too big too fast? Why ALL the problems recently??? You need to take a long look at how you’re operation is working and scale back if you can’t keep up.

My nginx web server stopped routing – I seem to be getting a 502 bad gateway error. However, I can connect to SSH. Seems quite odd.

This doesn’t seem like a small subset of customers. Not even a discount for this shoddy service!!

This is getting to be problematic. We depend on our email, this is happening way too often.

I take that back website down too…

Can’t operate a business this way…

I can’t stand this anymore. DH is just too unreliable. As much of a pain as it is to move, this unreliable service is a nightmare. Call it Nighmare Host from now on.

here we go again…. fourth time in a month…..

we’re down too! again!!!
please move ivolunteer.org to another server…

Can’t see mah’ sites!

We have clients depending on their sites to be up and running. Interestingly enough, all three servers we are on with you guys are apparently having the same problem. Doesnt seem like a ‘small subset’ to me.

Please make redundant mail servers!

This is unacceptable!

Eric: It just doesn’t make sense. Why enable comment sections on your official downtime blog if you aren’t going to monitor and respond?

They have no other way of informing customers of downtime.. you are expected to know of the existence of this site.

Either way I’ve tried submitting several “I want answers to all of the downtime and problem” tickets and they get ignored or passed off. There is no legit way to communicate with these guys IMO.

Sure, I can leave.. and it’s costly.. more downtime.. pain in my ass.. it just sucks.

The site is unreachable, but ping-able.

I don’t buy the explanation – I can’t contact the servers. So there is no connectivity problem with the outside world? This is crazy. I too will be researching alternate providers. After being a customer for seven years.

Can’t access any of my sites. Sorry to jump on the bandwagon guys but this is happening way too often for how much I have been paying you guys.

All sites on sanaa and porbeagle are down. webmail is down.

WAY to frequent with the downtime here Dreamhost.

Love your price – but moving to RackSpace is probably worth it.

How much am I paying for this service? There is way too much downtime lately. Your only job is hosting and you can’t get it right?

Someone tripped on the powercord. Plug it back in and get us back online.

Clifford.loa appears to be down as well… no mysql.
Noticed someone say they’re site got hacked not too long ago… Our joomla site did as well.. had all the correct permission set on the files/folders, but they managed to write malicious code to every file on the site.

Come on, get this resolved.

March 23rd, 2012 at 12:34 pmcantstandthisanymore Says:

we’re leaving dreamhost – goodbye

“I agree with several other people here, whatever is causing sub5 to REPEATEDLY fail, please change it. This can’t be just random chance, we’ve been down over and over. Simply put this is bullshit, and I am not alone in saying I want it fixed. I don’t want to threaten to leave, because let’s face it, that is a pain in my ass. You all count on it, so that I’ll stay, but I’m telling you now, I don’t want to leave, I want you to do your job and keep my services online so that I can do my job.”

@Jake, I have to echo these sentiments completely. We are a web design company, so this obviously throws a major kink into my workday.

While I can’t leave immediately, I think it’s time I begin thinking about transferring my major sites away and then slowly migrate the others…essentially leave DreamHost over time. It really does suck because I’ve been with you guys for easily 5+ years and have recommended countless others to use your service. But I agree, its just happening way too much now. If I don’t see any effort/chattter about fixing the homie5 issues, I will need to begin migrating away. :(

Totally bogus My Rails app stopped working on here last month and now these email outages. Why did I switch from Speakeasy (another awful provider!)?????

In the many years I have been doing websites- I give credit – the problems are not that frequent and theyu are on top of things generally. A recent site I had the oppertunity to work with took 2 ddays to get them to acknowledge they had an issue.
While it is frustrating, I find their level of support and response to be very good.

our mail is down. what timezone are you on? the latest post says 12:32 pm but it is 3:31pm EST. this post is to test the theory you are on PST. hope our email is back up soon.

MySQL server is dead. Affecting all my domainds :s, Please fix it fast fast fast!

Again… jesus

I have 3 sites with DH, 2 of them are down. Soooooooooo, i disagree with the comment above of it not being localized to a small subset of hardware.

Redundant mail servers would be good, but who doesnt use Google Mail?

Is there such a thing as redundant sql servers? THAT would be nice.

@Steven T

This website is hosted by Linode.

http://www.whoishostingthis.com/www.dreamhoststatus.com

Hi guys,

One person suggested Amazon is it a good webserver?

Regards,

Jimmy Deguara

I’m getting “Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf” when trying to get to my website. Crap.

Do dedicated servers on DH go down as much?

Mail deader than a doornail yet again … really getting weary of this. Inexcusable.

My site fails with a
“Warning: mysql_connect() [function.mysql-connect]: Too many connections …”
error.

Also, cannot seem to access the server admin (“panel”) page at all.

Not a good way to finish off the week …
/TJ

My site is still down – on WHAT WAS a big traffic day for me.

You’re killing me, Dreamhost.

March 23rd, 2012 at 12:37 pmBarry From Eastenders Says:

Replacement servers are on there way apparently http://bit.ly/GJUOnN

Did you pay your bandwidth bill DH? I wonder…

I must be in the “small subset” of your customers…..AGAIN. Email is down and my business is suffering.

I’m upset that we’re down too but for you folks who are going ballistic about this, let’s put it in perspective. Before coming to DH some years ago I went through a number of others where the service was absolutely horrible. We’ve got it pretty damned good here and if you think you can get better. Go spend a year somewhere else and then report back to us. (hehe, yeah, I know .. “as if”)

For the “you get what you pay for” types, you that you can pay 5 times the amount and get much worse service.

This IS the state of the art folks. Neither the internet nor the hosts that operate over the backbone are perfect. You’re not going to get any better elsewhere. Your anger isn’t going to buy you any happiness. Adapt to the technology or go Amish and get away from it.

And have a great day! :)

Mysql server is down too…..many clients are calling not really to say hi…

please let us know how long does this will last

I love how SO many people just bitch and moan about an outage. If you CANNOT deal with an outage once in a while to move to a host that guarantees a certain % of uptime and watch your wallet get smaller and smaller. I enjoy the tools that DH provides and it’s CHEAP so you won’t hear me complaining. Sure, it’s an annoyance but EVERY HOST will have issues, period! It’s just that DH TELLS you bout their outages and flat-out admits when it’s their fault. think Verio, NetSol or any other HUGE host will do that, good luck!

And if you do leave, good, less trolls for us that stick around have to deal with.

website hacked in December, repeated mail outages, now mail and site down

“This should only be affecting a small subset of our customers”.

Oh really? Seems that entire subset is here.

Arrrggg. when will this be fixed??????????????

Listen whether it be shared virtual or dedicated – is not the point now. I used to pay $199.95 for years and few issues. And now upping to $60/month and the issues have increased substantially.

Jimmy

LOL @ Barry! CLASSIC!

Yes, this has been a crap year, WTF y’all? We’ve had lots of down time especially on email in the past few months which crushes us.

I keep getting ‘Error establishing a database connection’. I’m guess that’s because of this outage. Seriously, this sucks. Please hurry up and fix this!

oh no! this is becoming routine.

sorry $19.95/month

I just lost $457,898,389 USD in business because of this!

Hhahahahahaha, waaaaaa waaaa, my server is down waaaa waaa hahahahahahahahaha

WHAT A JOKE DREAMHOST IS!!!!!!!!!!!!!!!!!! WE WERE DOWN 4 DAYS AND HAVE HAD SUPER SLOW SPEEDS SINCE THEN AND NOW DOWN AGAIN.

Kidding :P

Get me some nuclear waste so I can grow more fingers to keep counting downtimes!!!

I can’t access site. I see DH is down again. This is sad.

this is it unacceptable. Make it right. 5 minutes ago.

Mail/webmail has been down for quite a bit now, please get this resolved ASAP as it affects my ability to respond to clients and time-sensitive emails.

Thanks.

Seriously…who uses DH email anymore? Sucks to be down (though now my WP sites look to be back) but Gmail has proven the reliability champ over the past few years I’ve been using it.

Every time I see this I see small subset. I must be the Seina Yamada of the dreamhost world.

homiemail-sub5 is out – no webmail, no imap access from Thunderbird.

these problems always affect the same customers that I think is a bad service

March 23rd, 2012 at 12:40 pmstill don't mind DH Says:

can we put all the malfunctioning servers of all the folks that complain and consolidate them to one piece of faulty machinery, just to have a laugh here? cause we could use one.

“This should only be affecting a small subset of our customers” = DreamHost doesn’t care.

Service with this provider is getting flakier all the time. Sheesh.

March 23rd, 2012 at 12:41 pmBarry From Eastenders Says:

@Phadnis has such faith in dreamhost that he believes they can time travel…. hang on Dr. Eamhost…

can reach my admin panel, not working…
it was working fine 20 minutes ago, now it is not.

Pls correct asap, working on tight deadline to put together a site proposal for a prospective client,

thanks

mysql DOWN OH GOOOOODDD !!!!

Hummm, I guess 720 dollars a year isn’t enough to keep our website up… And there is not even a nice error page. Just read this, just 10% and we have to enter a ticket… hilarious:

“If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today! ”

http://dreamhost.com/web-hosting/

SMALL SUBSET MY ASS

Starbuck, I hate to tell you, but I fully disagree with you.

Dreamhost going down as often as they do is like RSA Security getting hacked and having all their tokens invalidated. Or like UPS stopping delivering packages.

All Dreamhost does is hosting and they do a ton of it. The one thing that they need to make sure is that their servers stay up and their customers’ services are available. That availability should be in the 99.999% range for a company this size. This is their core business and is what matters.

Customers should complain about this and customers will leave over this.

My mail connection keeps “timing out”
!

Sending of message failed.
The message could not be sent because the connection to SMTP server XXXX timed out. Try again or contact your network administrator.

mail service down!

I, can’t believe…Consider movingo to Media Temple, maybe!

MySQL is down. This is happening alot guys.

Bradon, do you know what shit you are even talking about?

Time for a credit to our accounts.

This is happening just too often.

anyone any ideas on how long did it take last time for DH to get the system back up and running? I may as well do home, its too late now to be in the office waiting for this to work. After reading most of the posts, I am seriously considering moving to another hosting company. Anyone again, ideas on whether this should take 10 minutes or 10 hours?
thanks

My website is still functioning but no email here. I understand that failures happen, but I’m another of the opinion that this is starting to happen too often.

Our mysql is still down – which means everything useful in our website can’t be accessed. Thanks for your prompt attention to this. We’re at almost an hour down time now. We had 3 hours downtime only a couple of weeks ago when we launched the new version of our site.

again? wtf is going on over there?

Seriously taking recommendations here for alternate hosting companies…

Pls not again!

I love the “Hey, yell into this cardboard box” method: Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

Totally unacceptable!!!! How am I supposed to run my business like this. I am definitely going to be looking for a new host. I have wasted the last hour because of this. I can’t respond to my clients or do anything right now!!!! It is totally affecting how I do business and the level of service I am trying to offer is suffering because of DreamHost!

jajjajaja esto ya es costumbre, dreamhost sin servidores caidos, no es dreamhost.

me rio por que ya me lo esperaba…

no se si reirme o llorar ?

Media Temple is double the price for Shared Hosting but they are seriously awesome and their staff is very knowledgeable/responsive.

http://mediatemple.net/webhosting/gs/

I think that’s my only alternative at this point.

Pleeeeeease my website is down. Quick fixes pleeeeeeease

I’m working with multiple DH accounts (not just domains) and they’re all affected, so either everyone in SF Bay Area got hosed, or the outage is not affecting just a ‘small’ subset. Quit saying ‘small’. If we’re on this page, we’re in that subset!!

Just registered a new domain today…well….what to show to customer? sorry to complain, but this is happening very often and people ask me why, so I do ask u why…

mail works for me right now

Serious? I have been so please with Dreamhost over the years. This is getting ridiculous. I want some major changes made and a refund for so much downtime.

I’m moving a customer’s email over to gmail right now over this very problem. Your service is simply no longer reliable for professional, stable email – just vanity domain hosting. Starbuck is incredibly wrong – the last few years have been terrible for many Dreamhost customers. I have to keep moving my customers off because of downtime and outages – it’s very frustrating.

@Mike G : Any failure is certainly “significant” but is any of this unavoidable without (exaggeration) infinite expenditure? I think DH learns from their errors but not every component can have complete redundancy. I used to get all bent about this stuff like our fellow customers here but all in all, this isn’t the Star Trek universe, computers are primitive, software sucks, and we’re paying a bearable price for bearable quality given the situation. IMHO, YMMV

Was the data center wired before the Civil War?

It seems like outages are caused by gravity… Look I know gravity can be hard to deal with when you want to fly but you guys have to figure out a way for it not to effect my server.

And then this

http://www.webhostingjury.com/reviews/site5 they are not as happy of late

Jimmy

their service status updates are awful

My computer is outside your network and I can’t reach sub5 mail server!

Well that’s cool. Lookin’ like an idiot for recommending DreamHost. I admit I was late on my bill, but it was pretty cool to drop a full payment on reactivating everything to a completely downed server. Everyone, name your to 3 recommendations for hosts to migrate to!

There was downtime last night too. Twice in fewer than 12 hours? It also seems that for some reason we’re always part of that “small sub-set.” Telling us it’s only a FEW people doesn’t make those “few” people any less aggravated.

cant access webmail. i need it today.

I feel compelled to agree with both Starbuck and Diesel. I own a small creative studio and the bulk of my business is website development and marketing. Having ANY downtime is irritating, but I have been with Dreamhost for almost 10 years and am very satisfied with the service I pay for. Admittedly there have been a rash of issues lately but I always try to keep things in perspective. Dreamhost allows me to offer a very good service to my clients at a reasonable price. On those times when I have had to explain outtages to my clients, all of them have understood an took things in stride (I have almost 100 clients with dreamhost at the moment).

But to each his own… if the downtime is too much for you, please by all means find another hosting company, that is what the free market is all about. But some of the extremely harsh comments I’m seeing are not warranted in my unsolicited opinion.

Dreamhost does the best it can and for me, that’s good enough.

Has anyone seen a small black cat with three grey hairs on the top of her head? I havn’t seen her since yesterday.

Just sitting here… who breathed on the server?… fire them!

I am a part of that “small subset”.

Site is down. Showing database error.
Thanks for getting us up and running quickly!

@James: It’s best is still worse than what I went through with Geocities in 1995. So, I suppose best is all relative.

I’m looking at Fatcow. I hear good stuff? Anyone else?

ALL VPS DOWN! please fix asap

I NEED MY WEBSITE ONLINE. IT’S THE ONLY THING THAT KEEPS ME AWAY FROM SUICIDE FEELINGS.

FIX THIS SHIT NOW!

3 out of 5 websites down.

This is unacceptable and much more widespread than you claim.

Merlinn Mann says SquareSpace. LOL

my mysql not work and http are slow, please fixit

what a nightmare… :( I have various sites and some VPSs on dreamhost… this year has been really hard for us

http://www.bluehost.com/track/tunatsub

here is my affiliate link for Blue Host if anyone is looking for a new host. Pretty good. Use good old C-Panel too!

What a joke!!!!!!!!

Is it just me, or did the 12:40 update say nothing that the 12:20 update hadn’t said? And speaking as a “small subset”, that line seems defensive and confrontational, as in “Hey, we didn’t take it down for our other customers, so I don’t know what right you have to complain”.

I’m happy you’ve worked another 20 minutes on the issue, but that’s still 20 minutes of downtime w/o resolution or explanation of why, precisely, it isn’t redundant or can’t be brought up on other hardware or at other sites.

What a great day to ruin my Friday guys: I’ve got 10 paying customers calling me right now demanding an explanation. As a bonus, several are threatening to fire me. Yay.

Almost 25 of my sites are down… Please help. Big issue.

“Error establishing database connection” over here!

last email inbox time code 11:26!!!

@James Sommer – I do agree and I normally am the one taking that exact perspective. I’ve been far beyond pleased with the service over the years, its really just been a recent sequence of events. Perhaps I should put a ticket in to support and really find out if they plan on doing something more to hopefully iron out the issues on this “problem child” server.

I don’t mind downtime, but I just want to know this isn’t going to keep happening, or god forbid, get even worse!

Guys!
just see how good services and performance it gives compared to other shit servers..
and other hosts often give more downtimes frequently each week..
i just completed one year with dreamhost last week and i didn’t faced a single issue till now.

but sometimes it happens like today becaz dreamhost is not gods creation! it is machine and machine do fails sometimes.
we should encourage dreamhost staff to get it up and running and not criticizing for what has already happened.

A HAPPY DREAMHOSTER!
Herit

Dreamhost i’m disapoint

“Small Subset” +1 – I wonder how ‘small’ the subset really is?

Been a dreamhost customer and fan for many years, but this is absolutely ridiculous.

@Mana… I completely agree. Everyone has a different definition of what “Best” means. I just wanted to point out that the vitriol seems a bit much.

Site completely down. Ugh.

Again, we have many problems…

We need our services ASAP

Jamie me too 3 out of 5 down. Have to go work on a new hosted by BLUE HOST….

I’m a pretty patient guy, and I totally understand downtime, but there has been an awful lot of it lately and it’s really starting to get old. Please fix the immediate problem ASAP, but then you really need to look at the underlying cause of the repeat issues we’ve seen lately.

I guess I am in the small subset. All my e-mail and web sites are down. I am in Canada so I am not internal and using my own email client.

Email is down again. The past couple of months have been the worst in 15 years with Dreamhost. Kick some a** and get it fixed for good, please!

That’s it. We’re leaving. Dreamhost is dead.

Hi!

We are using CGI Perl, just like you do on DH :-)

However, we are getting peculiar results on our frontpage on http://www.morsmal.org I hope this error does not result in corrupting our db files…

Otherwise, I think that DH rocks. We have been customers of DH for over 10 years now.

On

berbils (ssh/ftp) server also appears affected.

Now it’s not just our MySQL but the site as well – even the homepage isn’t loading.

Now my FTP stopped working…WTF…

In 2012 I had problems with sending e-mails.
I had not had before.
It is necessary to hire another service or migrate to the mail server?

Domains are down on two different VPS accounts here.

C’MON!!! Not again!

@Lars.. Hi Lars, I agree with you regarding the current rash of downtime. I’m hoping that the otherwise excellent DH team will get this issue taken care of and then we’ll be back to having great service for a long time. :)

How can I switch from Dreamhost to another server when Dreamhost server fails? Is there a way to do it via software?

All my websites are down. They are at “carina” server.

I’m guessing a lot of people are just posting here and not reading. I’ve made a couple comments saying we just need to weather out this situation and I tend to stick by that.

But…. it sure looks like this isn’t a little rain falling, but a huge storm…

Ya know, for just once, next month I don’t want to see another newsletter with 95% shtick and 5% content. If things are getting this bad over there, DH needs to get a little more professional for a while.

March 23rd, 2012 at 12:57 pmRotten Business Spellers Says:

I’m amazed that the numbers of “professionals” posting on here complaining about professional services being down that can’t seem to spell enough proper words to make their professional points. Part of being professional is having communication skills (yes, certainly DH could use more of that on this blog, absolutely), but I always wonder if having all these unprofessionals involved is tainting our shared hosting services. Do their hobbyist website offerings attract hackers or DDOS to dreamhost? Would you, if you were DH, want these customers screaming such misspelled rants at you? The real professionals either expect occasional internet hiccups, run their own services, or they just get up and leave quietly.

For those asking about other hosting.. I have an ecommerce website that I host on a dedicated at codero.. about $180 month.. Its been 2 years since the last outage, and that was a DDos attack, which they thwarted in just a couple of hours.
I’m only here because I’m developing a WP site for a customer and thought I had messed up the DB connection in the config file… The frustration everyone is feeling at this point with DH is justified.. There needs to be a better plan in place for these things.

The latest status says only internal communications should be affected but my site is dead in the water. Probably because its a shipping site and requires access to mysql. Problems have been coming often lately.

Mcafee won’t even certify my site anymore because they say the operating system is no longer secure. I told DH and they said, politely, “yes, we know but we have no plans to change.”

We have been Dreamhost customers for at least 5 years, and I have never had so much downtime like in the past few months. My site is down right now, yet again. What did you guys change? Your service was not spotty like this. We used to recommend it to all of our friends. Now I’m starting to think we need a new hosting service.

Too many problems in a few days, I think DH must check the whole problem. My clients are questioning about so many downtime. and we start thinking about changing supplier….
DH, we count on you…..but time is money!!!

YEA YEA YEA NO SHTICK IN THE E-NEWSLETTERS FOR F SAKES!

I encourage you guys to post comments Facebook.com/dreamhost

They are clearly more concerned about getting new customers than focusing on these constant downtimes.

This ALWAYS comes at the WORST possible time. Media Temple looks like it might be the way to go folks. I was a standout, hoping the problems were just a blip, but it seems this is their modus operandi. Christ, I do not want to have to move everything.

You should have re-routed the traffic long time ago.

Sorry no seu!!!

Come on now!!! What is going on!!!!

3 times sad!
– from Brazil

We are currently down at http://www.ethoseo.com

Hope we’re back up soon.

WTF! Again? And “Sorry…” is all you have to say?

En 2012 he tenido problemas con envio de correo electronicos.
No habia tenido anteriormente.
Es necesario contratar otro servicio o migrar a servidor de correo?

webmail.compuproveedores.com
webmail.hpstorehermosillo.com

We’re also considering moving to MediaTemple, too much problems lately… hard to explain to customers. Maybe we’re just unlucky and always end up in the “small subset” with problems.

2nd time in a month. This is thoroughly unprofessional!

How much time do we need to waiting?

My website is down too; we get this message: “Could not connect to MySQL: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 110″

website down as well now…. the full set. well done dreamhost!

I think you all should take an interest in astrology. We’ve been going through a transit since February where machinery and technology will break down. This is all universal work :)

It will clear up next month.

Namaste.

Not a soul has seen my missing black cat?

site and email down now, im dying here come on..

Anyone have feedback on ipage as a hosting company?

In 2012 I had problems with sending e-mails.
I had not had before.
It is necessary to hire another service or migrate to the mail server?

webmail.compuproveedores.com
webmail.hpstorehermosillo.com

i migrate to a VPS because they said it was more stable, but problems remain. will dreamhost compensate me with something?

My VPS was working fine after I changed the DNS servers to google’s 4.4.4.4 and 8.8.8.8.

But now the VPS is totally inaccessible (possibly down), unresponsive to ping.

What I don’t understand due to the state of technology today, how this can happen? It’s as if all of our stores are closed and we’re certainly not getting compensated for lost sales.

At this point, you do not have an automatic alert system that just re-routes traffic? What is going on there at DH. I know that it doesn’t take that much to re-route if plans are in place, or are you now in another country where perhaps there has been electricity outage? Come on. This is really unacceptable.

so uh… what happens to all the emails I’ve written (and replies I’ve been waiting for) when something like this fails?

Servers down for http://www.coldfusionnow.org.

Can’t connect! And today the anniversary of Drs. Fleischmann and Pons’ historic announcement.

Please hurry Dreamhost!

Ruby Carat

PS. Even before that, I could ping DreamHost’s DNS servers; they just didn’t respond to DNS requests.

again mail service down!!!

Webmail down for me. I have been with dreamhost for years, but this is stupid. Dreamhost has major outages at least once a month. Disgraceful. I’m going to switch to google mail.

March 23rd, 2012 at 1:04 pmBarry From Eastenders Says:

@Brett this is closely related to what you just said http://bit.ly/iCof9S

Justin, the mail servers which will be delivering mail to you can be expected to retry delivery for several days before giving up and bouncing a message to the sender.

Is the network team makings things worse?
I was able to access one of my sites, then the db disappeared, now I can’t ping it.

My first domain that went down has stayed down the entire time.

>This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.

Website and mail are down! I’m so stupid.

Everytime this happens at dreamhost, I figure it’ll be the last time, and I don’t migrate. And then it happens again. The joke is on me!

I’m thinking network solutions, maybe slice host or something — give me anything reliable. At twice the cost or even three times the cost per month, I’d come out ahead, given what happens when we go down.

Geez, AGAIN?

Count me as 1 of the subset…site is down. Even if 95% of your sites are up, that means nothing to the 5% that are down. I understand with technology nothing is perfect, but, are we learning from the problems and are we making plans to avoid them in the future?

Last time this type of issue happened, it lasted several DAYS, forcing me to switch my outgoing mail to an SMTP server from Comcast. Now, my outgoing is fine, but no incoming mail works. Looks like the only option is to stop using DH for basically everything.

Starbuck: Things are getting pretty bad.. in my 6 years here I’d have maybe a once yearly downtime, that was annoying, but understandable.

This is the fourth downtime in 2 months now. And 2 of those were for over 8 hours in length.

I’ve also had some other major problems. In particular, MySQL memory usage suddenly jumping unexplicably, which DH blamed me for, and I ended up having to pay extra for. After I did everything I could on my end I sent them several support tickets, and they repeatedly told me it was my issue, without doing anything to try to help me solve the problem. It’s not like I can do much on a MySQL VPS either.. you can’t even reboot the server through the tools, only the MySQL service on the server, something they changed a few years back.

After one of the above downtimes, and about 5 months of paying an extra $40 a month for memory I swear I didn’t need? The memory issue magically disappeared. Clearly something they did on the server end to fix the issue, and when I posted about it in DH comments? Several other people stepped forward to claim they were having the same issue that also disappeared.

I have not had the “stellar service” the DH apologists claim they get in these comments section. I’ve had support tickets just outright ignored before, gotten a form reply telling me to send another ticket at another time if I am still having an issue.. they didn’t even bother to read my ticket. I was just too pissed to even send another one.. but luckily, weeks later, the problem appeared to resolve itself.

March 23rd, 2012 at 1:05 pmMansoor Seikh Says:

This is unacceptable from a service providers who advertises 100% up time. Please fix it.

I’m not kidding, I see a lot less downtime and connectivity issue with http://www.000webhost.com, who provides completely free service. Its limiting in many regards but at least keeps the site running.

I HAVE NO REASON TO LIVE WITHOUT MY WEBSITE.

It’s not just affecting webmail. Whole sites are down, even static HTML sites…

Wow – I launched a set of six books today and no one can access my website for more information. No one can email me about them. I’m getting very tired of all the down time. I think it is time to migrate somewhere else.

We’ve been down for hours at this point!
Seriously NOT COOL!!!

We’re currently covering a music festival at http://forgetthebox.net people are coming to our site for festival news, show times and lots of other stuff

Grrrrrr!!!!!!!

Jackalman ftp and http are down.

I can’t believe that all 20 of my sites are down.

@anon – I hear ya.

Im barely recovered after the Feb23 to March ? outage on stampeders and now this?

What another outage. This is ridiculous . We are looking to move to Amazon ec2, this is absolutely BS with the chronic down times!

Episode 7 of my podcast “You Talk Loud” (http://youtalkloud.com) is to be released and it’s being delayed by this. Listeners are pissed!

AHHHHHHHGGGGGGGGGHHHHHHHHHH!!!!!! NOOOOOOOOOOOO!!!!!!!!!!!!!!!

This is REALLY getting old.
Please guys…FIX THE PROBLEM!
I really don’t want to have to
leave Dreamhost, but this is
getting to be a serious
problem…

this is getting old, lots of problems in 2012. Even when working my sites seem super slow to load

STILL???????

I was just getting ready to destroy the one ring to rule them all when suddenly my website went down. Now I can’t e-mail Frodo and the rest of the Fellowship. I’ll have you know that if Sauron wins this battle for middle earth, it’ll be the end of Dreamhost once and for all.

And don’t forget to never utter “You shall not load!” while crossing beneath the Misty Bits of cyber space.

I’m afraid I’m going to have to switch RohanHost.com if this isn’t fixed. Horse’s could do a better job.

Que horrivel!!! Mais uma vez isso acontece…

MY BLOOD IS IN YOUR HANDS, DREAMHOST.

fomalhaut is down. (My web server)

I remember the forums going nuts like this at PowWeb, YourHost, and GeoCities. Mass migration isn’t the answer. I think the customer base needs to do something to keep DH Support and management accountable for this, working together rather than taking the knee-jerk solution and bailing.

please fixit now, i cant work

I wonder what service dreamhoststatus.com is on, I want to move to that server!

My client’s website http://www.monkeyhousemusic.com has been down for a while. Apparently I am in the ‘small percentage’ of customers effected. Please fix it. — Steve

Take that you stupid Wizard Gandalf!

I knew that once I infiltrated your hosting company’s payroll it wouldn’t take much for a couple of orcs to unplug some things and take down the whole network.

You can’t fight Sauron, Gandalf! You can’t!

Now hand over the ring and I’ll think about letting your website come back up!

no mail no website
all down!

this is getting to be monthly nightmare if not more. Very frustrating. Been with Dreamhost for ten plus years, now looking elsewhere. Too bad.

March 23rd, 2012 at 1:15 pmAndrea Jones Says:

And I only switched from hostgator last night (was a temporary move for one month while they sorted out the spam blocks)… think ill be moving back

Lots of problems this year at Dreamhost. Anyone would care to suggest another company worth migrating to?
Tired of this.

@rabs Do not be dramatic loser.

March 23rd, 2012 at 1:16 pmBarry From Eastenders Says:

I am certainly of opinion that genius can be acquired, or, in the alternative, that it is an almost universal possession. Its rarity may be attributed to the crushing influence of a corrupted society. It is rare to meet a youth without high ideals, generous thoughts, a sense of holiness, of his own importance, which, being interpreted, is, of his own identity with God. Three years in the world, and he is a bank clerk or even a government official. Only those who intuitively understand from early boyhood that they must stand out, and who have the incredible courage and endurance to do so in the face of all that tyranny, callousness, and the scorn of inferiors can do; only these arrive at manhood uncontaminated.

HOW ABOUT AN UPDATE??????

Wow! The comments are so cold! What’s wrong with you people? DH is having issues and it’s bye bye see you later you suck?! Grow up. Sh*t happens. Go outside once in a while. Read a book or something. DH, I feel for you. As far as I’m concerned you’re still a great company. Thanks for all your hard work.

My email is down, my site is down.
I look bad with my clients.

no website no email none of the nothing AGAIN!!!!!!!!!!! Please can someone post a new hosting service that we can all move to I am sick and tired of this BS

FTP and MySQL down. Again?

March 23rd, 2012 at 1:17 pmBarry From Eastenders Says:

@Dave I HAVE LOST MY SKITTLES, THEY WERE HERE A MINUTE AGO!

Hey Dreamhost there is this special thing called redundancy, you might want to look it up. Unreal

Dreamhost, I really like your hosting plans and your service, and I appreciate that you’re transparent about the problems with the servers. Thank you.

BUT PLEASE FIX THIS!

Nice to know that I’m a part of “only a small number of customers who are affected.” That small number of customers is 100% affected.

This is the third time in the last month (that I know about). I do hope you guys are thinking about how to address this issue on a wider scale than just rerouting the traffic.

My Hemodialysis system that runs on dreamhost is down!

PEOPLE ARE DYING!

@cleiton FUCK YOU MAN! MY DOG DIED. MY GF DUMPED ME FOR A DJ AND MY BLOG FRIENDS HAVE BETRAYED ME. I DONT HAVE MONEY AND MY MOM DONT LOVES ME. I’D RATHER DIE THAN BE WITHOUT MY WEB-FRIENDS.

HAS ANYONE SEEN MY CAT!? There must be 10000 people posting, nobody has seen my cat?

what the ever loving fuck, dreamhost.

I use and recommend hosting by ZIPPERZ.NET. Check them out and tell them Kevin sent you! http://www.zipperz.net

BTW, I’ve been with Everyone.Net for email for years and highly recommend them. I don’t trust shared hosts for email. Now I’m second-guessing trusting DH for MySQL too, but hell, you have to trust someone, and distributing the topology too much just increases vulnerability. *sigh*

March 23rd, 2012 at 1:19 pmBarry From Eastenders Says:

@Marco lulz

Thats it this is the last straw.

March 23rd, 2012 at 1:20 pmMarcelo Oliveira Says:

everything stopped, email and website

Everything is down, website, email, everything……… Not good

Im hosting a porn site on dreamhost and people need their fix! Please hurry.

@Paul Huang haha

From the Hitchhike’rs guide to the galaxy:

Far out in the uncharted backwaters of the unfashionable end of the Western Spiral arm of the Galaxy lies a small unregarded yellow sun.

Orbiting this at a distance of roughly ninety-eight million miles is an un utterly insignificant little blue-green planet whose ape-descended life forms are so amazingly primitive that they still think digital watches are a pretty neat idea.

This planet has–or rather had– a problem, which was this: most of the people living on it were unhappy for pretty much all of the time.

Why? Because they used dreamhost

>there is a small subset of our network which is unresponsive,
I have been with you guys for a while. Is it why I am stuck in a old server and fall into this small subset? Can I move to a a newer server in a more modern environment. This is worst than the standard 99.5% up time…

Interesting how the dreamhost status is up but the customers are down.

THE WHOLE INTERNETS IS COLLAPSING!!!!

March 23rd, 2012 at 1:21 pmBarry From Eastenders Says:

@rabs has your CAPS LOCK GOT STUCK AS WELL??? That would suck more than all of those other things that HAPPEN IN CAPS LOCK WORLD!!!!!!

This is the 3rd time this month. No email, no website means NO MONEY. Am I getting a credit for all the outages? This is garbage.

hey all my sites ftp are down…. need some info in there :S

@rabs you deserve even worse you fucker. you’re such a loser. I think you should really die

The frequency of service outages has been increasing over the last few months or so and the service outages take at least an hour. Already my overall traffic is down and a significant factor is the outages. I don’t want to move to another host (because I don’t have the time) but I don’t think staying here is a viable option.

It seems I’m one of the few affected customers… =(

Um, I was expecting a really important email today. Does this mean that email is just “lost” now? Or simply delayed? Are people who email me getting error messages? Because that is going to look REALLY BAD when I was up for a job interview. :\

This is getting to be ridiculous, everytime my website goes down for 4-8 hours i lose an entire working day.. Your downtime is not only causing an inconvenience but also costing me money which is needed to run my business. If customers can’t shop my website, they can’t order anything.

Time for a new host, too bad i paid for 2 years upfront last year..

Hey what service are you using to keep this status page up? I need their number!!!!!!

Justin – maybe your business doesn’t depend on having a web site up and received and sending email in a timely manner. The job of DH is to keep it up. Really. Downtimes have happened more lately and I have to bear the cost of it. Why should I have to explain to clients and prospects that I contract my hosting to a company that fails? Hello!!!!!!

wesites and mail (pop and webmail ) still down.

not impooving.

This has been going on all afternoon… email down, website(s) slow/partially down – and how many times in the last 2 months? This is inexcusable for a hosting company, esp one named “Dreamhost” – this is more like “HeadacheHost”. Sounds like you need more reliable hardware, need to fire the head of server purchasing and maintenance and move along.

I think it’s time we start suggesting other, more reliable hosting companies… anyone? I was with Siteground but they fell apart too. SimpleURL.com may be what I return to.. never an issue there. Other suggestions?

Sad.

Annoyed.

Perplexed.

I see no improvement. Time estimate? Trying to decide whether to send an e-mail blast to all our subscribers…

Improvement not noticable, but lameness is.

LMAO at some of these comments.
I can’t take the drama here anymore. I need to keep my business running despite the failure of some components.
See you on the other side of the event horizon, folks.

At least people’s sense of humor servers aren’t down :)

All of my sites are down…. meissa.dreamhost.com cannot be contacted at all….. this is the first time (that I’ve noticed, at least) that this has happened since I started with DH but it’s certainly a bummer….I was planning on putting two new sites up today :-/

Well, all sites are down (about 10) and all services too. Jabber, mail, databases, webserver etc…

I’d say it’s not just a small subset and it’s not just a few services. Everything seems to be affected.

This is the second time in three weeks this has happened. Unacceptable.

My sites are down completely and it is a busy time for one of them as it has details of a night I am promoting on which is happening tomorrow so this is pretty bad for me as a customer. Please put it back online soon :(

http://distant-planet.org

Site and email down, it sucks….

THINGS ARE NOT IMPROVING. MY SITES ARE STILL DOWN.

My site is still down.

March 23rd, 2012 at 1:26 pmBread Muncher Says:

Trying to make a sandwich but my site is down. WTF

If by improve you mean not getting any better… then yes tons of that over here.

Why is there always issues within this cluster????? I think everyone would be pleased if you move all Homie to new hw platform!

Has DH gotten uglier since the new CEO started? Perhaps they are ‘optimizing’ their service.

But whatever… The upcoming newsletter will be funn-nee!

This is technology we are talking about.
Things fail. CALM DOWN, CHILL OUT, and take a breather.
Things CAN’T ALWAYS be up 100% of the time.
Technology fails sometimes, and they are trying to fix it.

@Barry From Eastenders I DONT REMEMBER ASKING YOUR OPINION. GO EAT YOUR CEREAL AND JUST LET ME DIE.

You know what I really need?

Turkish oil wrestling.

Winner’s on bottom.

Not so sure about the things are improving part. Doesn’t look like much has changed as far as I can tell.

I’ve been a DH customer since 2005. This last year has been completely different from the previous 6 years I’ve been a DH customer. I really love the DH control panel and company culture, but lately I just can’t count on the service. Plus the CS Reps used to be nice, helpful and informed and lately they have been lazy, rude and don’t seem to know what they are talking about.

What the heck is going on?

I want to move to another host but I don’t want to use CPanel or Godaddy or Rackspace (too expensive) and I don’t especially like Media Temple though I’ve considered that as the best option. I really dislike 1and and HostGator.

Any recommendations for a good, affordable host whose control panel is as nice as DHs (doesn’t use CPanel)?

Now I’ve gone from having no MySQL and email to also having no website or ssh.

Still down……email and site…………..Unacceptable.

Please buy some new hardware…and new routers…and whatever else you need to keep this from happening again.

Awwwwwh. You might suck ass sometimes Dreamhost, but I still love you.

Our site is still down. This is getting old.

Oops, my websites just went bye bye too, not just MySQL, Jabber, etc.

Oh yeah, I forgot, the world is supposed to end, this is 2012. The Mayans had it right all along.

Justin says: “Wow! The comments are so cold! What’s wrong with you people? DH is having issues and it’s bye bye see you later you suck?! Grow up. Sh*t happens. Go outside once in a while. Read a book or something. DH, I feel for you. As far as I’m concerned you’re still a great company. Thanks for all your hard work.”

This has been a paid announcement. That is all.

Well, things are improving now. Instead of very down, all is only normal down.

Gracias por su pronta respuesta equipo de dreamhost.

“Traffic is now being routed around the problematic section of our internal network.”

My site is completely down. That update from above is not true for me.

LOL. Homie is fricken terminal. Need to get Kevorkian to put his ass down for good.

hmm my site’s still down :|

Bad, service… webmail, web, down… again! Shit

Justin says: “Wow! The comments are so cold! What’s wrong with you people? DH is having issues and it’s bye bye see you later you suck?! Grow up. Sh*t happens. Go outside once in a while. Read a book or something. DH, I feel for you. As far as I’m concerned you’re still a great company. Thanks for all your hard work.”

You really are a complete fucking moron!

March 23rd, 2012 at 1:30 pmTech Wizzird Says:

That’s it, screw you guys, I’m leaving this place, and taking all my hosting business to Dreamhost!

That’ll teach em a fucking lesson!!

MySQL server is dead.

Base64 decode hacked again, at the same time as this “Data Server Connectivity” issue.
Pretty much destroyed my blog with over 3000 posts. Congrats.

A “Small subset” of customers. That’s irrelevant to me as a member of that subset. That does NOTHING to ease my frustration.

Our site is still down as well. No connexion to the database yet.

March 23rd, 2012 at 1:31 pmBarry From Eastenders Says:

@Rabs That is just awful, i mean truly horrible…. to allude to cereal eating on a public forum… shame on you for outing me like that in public. Dont tell @Bread Muncher I said I would make him some as his sandwich server is offline..

@cleiton why are you so mean to me? i just want to be happy.

All my sites and email are down. Can I expect a refund for the downtime?

Still no service. No improvement. Just completely blank.

I was fine for first hour or so of this and now hagtna is down. How exactly are you managing to make things worse?

March 23rd, 2012 at 1:31 pmBarry From Eastenders Says:

@Michael…. in china town

GUYS,
we need a status update my site is down since about an hour ago no email, no site nothing. This is seriously it I’m done with DH i pay far too much for this crap to happen every month. What wrong when is it going to be fixed.

I think they’re still a great company. I just hope they’re learning from this rash of recent outages. Time will tell.

Initially just webmail wasn’t working for me. Now I have sites down too.

March 23rd, 2012 at 1:32 pmBarry From Eastenders Says:

holy hell @Rabs found the Caps Lock…. we’re DOOOMED!!!

Why is it that website always seems to be in that “small subset of customers” that are effected? The last 6 months Dreamhost has had way too many problems. May be our last year here if things don’t improve significantly.

My site is down, can’t get to it and neither can my readers. This happens way too much in my opinion. I have things I need to get done and not being able to get to my site is not good at all. I don’t need to be losing readers because of this. I think it’s about time I find a more reliable host.

DH – sorry you’re not looking good in my eyes anymore. cute emails don’t make me feel better about lousy uptime.

My site’s on harare and is totally down. (can’t ssh in) I can’t seem to reach mysql server either, though I can’t seem to figure out which one i’m on – perhaps it is the “spurdog:dobrich” mysql server?

Almost ready to switch to hostgator!

Come on people! I am trying to run a business and these perpetual issues are really starting to stretch my patience and understanding.

Same router at 66.33.201.113 is still giving “Destination Host Unreachable”.

Sure doesn’t seem like anything had been “routed around”…

Update Mar 23rd, 1:18pm PST >>>> B.S: all my websites are DOWN-!!!!1!!!!!!!

ROUTED MY FUCKING ASSSSSSSSSSSSSSSS

Websites are still down here. You know that “small subset” of customers are the ones reading this thing.

Come on! All my sites down again! At least 4 major downtimes in the last 2 months… what a hell is going on!

I agree with Justin… shit happens. And in the past, we’ve been highly patient and understanding of down times. And appreciative of the work it took to get us back up and going. But the last three or four weeks have been exceptionally bad. Our site getting hacked… multiple times… email passwords getting deleting magically (with one of our emails we have had to reset with a new password every morning, and twice in one day)… inconsistent access to email… and today our email has been down more than it has been up.

We are a small non-profit and have to watch every penny. We appreciate getting free hosting from DH because we are a non-profit, but just in the past three weeks we have had to pay more money to get our site un-hacked than we saved with the free hosting.

So as much as I hate to say it – or do it (as we have to pay someone to migrate our sites to a new host) – we have to explore other options.

LOL… It sucks that my Companies DH server is down, but these comments are very entertaining…

March 23rd, 2012 at 1:36 pmBarry From Eastenders Says:

@Josh You did what? Well, each to their own I guess….

I’ve been getting a total outage: Network is unreachable for the past 40 minutes.

Before that it was just MySQL outage.

So it’s been getting worse.

March 23rd, 2012 at 1:36 pmChris Madison Says:

Again another f”ing cluster of bull. Gotten to big and now a lumbering beast. Down with the beast I say!!!

Our website is down entirely. won’t load at all. For more than an hour?

@rabs You do not deserve to be happy, you must have a small stick, be fat and living with his grandmother.

I just want to eat my butter quietly.

Site down again ?

eat a dick dreamhost

the last 2 years 4 time down!!!!this is terrible……..

Oddly, none of my sites or clients’ accounts are down, but for the last month anything LAMP has been slow as molasses. Even wordpress sites. Sheesh. In the last few days, I’m getting gateway time outs for administrative pages in most of my drupal sites over multiple client accounts. Of course cached pages render faster, but they’re still sluggish.

I just found out yesterday that Cron hasn’t worked since January for any of my clients’ accounts. Several accounts no longer have access to ordinary bash tools like ping, traceroute or ab, while others do.

I can’t get on the forum to ask questions there because the password recovery tool is broken.

Pages shouldn’t ‘gateway timeout’.

I feel like there’s llamas eating my heart. @Barry From Eastenders, please, i’m really hurt.

Well my SVN Access is back up (For now)

@Will “Small subset of our customers” is runaround speak for “Everybody on the Homie server”

really really… again gonna cancel asap

@cleiton MY GRANDMA SAYS MY DICK IS BIG! YOU DONT KNOW SHIT ABOUT ME!

March 23rd, 2012 at 1:38 pmBarry From Eastenders Says:

@Rabs awwww fake Rabs, I want CAPS Rabs back!!!!!

Why is it always the Homie server is that thing a samsung potato or sumething

Please provide an update! It’s been nearly an hour and a half since the last update. This is unacceptable! Researching new hosts immediately. I can understand outages, but this is terrible customer service.

no SSH for me, too. web site works fine, though. but SSH is what i want.

please help the people pay yours jobs

Well, ALL of my sites are DOWN…. and from the other posts, I wouldn’t consider this a SMALL Subset, ESPECIALLY since email and SQLServer are impacted! I know DH doesn’t even look at these posts, so threats do no good. But this is RIDICULOUS!!!

@rabs you’re a fuckin betrayer. u deserve burn in motherfuckin hell, son of a bitch! You r a traitor, a damn traitor! u betrayed ur friends, YOUR FRIENDS! i cant look in your face anymore!

STOP SAYING THIS ONLY AFFECTS A SMALL SUBSET OF YOUR CUSTOMERS!!! Jackasses, for those of us who are DOWN it is a big deal. Stop trying to minimize the problem.

Everyone needs to chill out. The world isn’t going to end because your email and/or website is down for an hour or two.

Dreamhost: Thanks for the communication. Hopefully you get this resolved soon. Add a 6-pack of Bud Light to my hosting bill and send it Brandon’s way. Thanks for the hard work.

Can someone please give us an estimated time when our websites be live again? Thanks.

Guise, does anyone have a sample of a file that got base64-hacked? I’d like to know how those look like to know if my files also suffered that.

Email down still — all day. East Coast

Hey Dreamhost — is there something other than Squirrel Mail we can get from you? I’m a non-tech just trying to read some emails and such but in the last month or so it’s been down 3 times or more for extended periods.

thanks

Ryo: try the support or wait for updates… as stated above: “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

Perfect! Right when I’ve promoted a web registration… the site goes poof!

BACK-UP as part of a disaster recovery plan sound like a good idea? Especially with multiple customers on a server – shared or not! It’s still your business. Wow.

@cleiton this guy is a piece of shit! a dumbass! he’s not worth a damn thing.

@Pedro Tulio GO STICK YOUR ASS WITH A CACTUS. YOU GUYS WANT CAPS? ILL GIVE YOU CAPS!

EVERYTHING IS DOWN !!! MYSQL DOWN !!! FTP DOWN !!! ALL MY WEBSITE IS DOWN !!! DREAMHOST ONE OF THE LOUSIEST HOSTING EVER.. I REGRET I SIGNUP WITH THEM.. TOOOOOO MANY PROBLEMS ALMOST EVERY SINGLE DAY.. THEY LOVE TO MAKE MONEY BUT GIVING US UGLY SERVICE.. DOUBLE STANDARD.. SLOW PERFORMANCE AND USELESS TECHNICAL SUPPORT.. WHAT EVER YOU ASK THEM, THEY WILL KEEP SAYING THIS N THAT OR SIMPLY GIVE YOU THE LINKS TO THEIR OUTDATED WIKI DOCUMENTATION.. SUCH A UGLY SERVICE AND UGLY FORUM, TOO LAZY TO ANSWER THE QUESTIONS.. IF I AM LIVING IN THE USA, SURELY I HAVE ALREADY MOVE TO ANOTHER HOSTING COMPANY WITH A POWERFUL DEDICATED SERVER.. BUT UNFORTUNATELY, I AM FROM MALAYSIA, SO WE HAVE TO PAY 3-4 TIMES HIGHER THAN THE ACTUAL AMOUNT.. ANYWAY, HOPE THIS PROBLEM WILL BE SOLVED..

I HATE YOU!! I HATE EVERYBODY HERE!!!!

CAPPPPPPPPPSSSSS LLOOOOOOOOOCCCCCCKKKKKK FFFFFFFOOOOORRRRRR TTTTTTHHHHHHEEEEEE MMMMMOOOOOOTTTTTTHHHHHHEEEEEEERRRRRRR FFFFFFFFFUUUUUUUUCCCCCKKKKKKKIIIIIIINNNNNNGGGGGGGG WWWWWWWIIIIIIIIINNNNNNNN

Bangkok server is down completely, as far as I can tell, no FTP, no HTTP, MySQL, nothing.

Not sure what Brandon means by things are beginning to improve. I certainly haven’t seen any improvement and apparently nobody else here either!

Yikes. I’m using homemail-master for some clients.
Is domination mysql server in the homie cluster?

“Error establishing a database connection” is all my WP sites show. How nice.

I’m really glad to hear that this is only affecting a small subset of your customers, though. (Yes, that was sarcasm.)

So….If I’m part of this small subset, I do pretend somehow to be paid back for this; I mean….I was just showing a customer how solid my services could be for his busynesss….and now he is obviously gONE!!! PORCO DIO

March 23rd, 2012 at 1:45 pmAbdul Kebabdul Says:

My Al-Qaeda terrorist site is down. Please fix it.

OMG, I had a lot of my clients switch to you and now they’re all down, this reflecting really poorly on my business and my clients and I are all losing money today.

I was with you for years without a problem, and in the last year there has been numerous issues, please get your shit together, don’t make me switch hosting again :(

Is there a hosting plan on here that doesn’t have problems? If so, I’d like to know so I can switch to that.

My sites are all up. MySQL working. Just my email is down.

now my web site is down as well. thanks a lot for wasting another day of mine

From which hour the server is down? I didn’t realice till now, and people have been calling me for this isue.

I’m down too, both legalectric.org and nocapx2020.info. At least I now know it isn’t the utilities I’m fighting shutting it down! Will email be bouncing or ???

My site is down, so its not just email… good luck with the fix guys; its morning here in Australia and people need to see my site!

SQL appears to be back up for us, email still down.

Data: Lal is undergoing cascade failure, Captain… I can’t save her.

Couple of sites and MySQL servers are back up, but nothing for email.

We’re all down!!!!! Please help!! VPS’ and all. :(

Read this article: Setup a Dreamhost-independent Failover system

http://wiki.dreamhost.com/Failover

20 users is considered small, when its 400+users its no longer considered small.

who knows that their definition of small is but it needs to be redefined.

@Rabs you was a brother. you was my friend. how u can sleep after u did to me?

MY WEBSITE IS BACK!! I HAVE A LIFE AGAIN!!!!

JUST CAME BACK TO LIFE

@Brandon

“working diligently toward a resolution”

……. I’D SAY YOU BETTER GET WORKING ON IT A BIT MORE QUICKLY. DILIGENCE?!?!??!?!?!?!???!!?!?!?!

my sites are back up. im with the “ive been with dreamhost for years, i like you guys. but get your s*** together group”.

anyway, back your sites up now boys and girls.

@Craig – I’m not sure if this is the case for you, but if your organization is based online, and your livelihood (and paycheck) depend on getting things done online, and a post has been parked for almost 3 hours now? That’s an issue. If dreamhost had thrown stones through your store front window, made unusable your register, and pissed all over your customers, wouldn’t you be a bit aggravated?

I NEEZ THE EMAIL WORKING!!! PLEASEEEE

I’m back up… Thanks.

The fact that you say it only effects a small portion of your customers is not very comforting to those of us who are down. We are trying to conduct business and there are only so many hours in a day. This has to stop, we cannot conduct business properly with as much down time as we have had the last 6 months

Sites are up for us! Thank you so much for your help!

Such a bummer – my client is really angry. And DH is not calling me back about it though I’ve paid for the premium callback support. I am looking into other hosting options. It’s so embarrassing because I sold this client on DH and they trusted me and let go of their old hosting company.

Now I just sit and wait, spending my Friday afternoon in limbo land……

Sending DH lots of love and light for a successful recovery.

@Pedro Tulio I BET YOU HAVE HAIR ALL OVER YOUR BODY. I HOPE YOU BURST IN FLAMES, SINNER.

These issues are discraceful that we have been experiencing with dreamhost lately im at the point if this isnt resolved quickly or if it happens again soon i will consider moving my hosting to another company. can anyone recommend anyone else.

Regards Luke

THEY’RE BACK!!! YEAAAAAAAAAAAAAAA!!!

i´m still down, saggita and mysqlserver.
Esta dando que desear su servicio, yo sabia que tenia que migrar en febrero a otro hosting.

site up, Still no email. need suggestions for new service.

FATCOW.COM

Email STILL not working 1:46pst

Okay, we’re back now. That was just under 2 hours of downtime.

How come anytime there’s something down that only affects “a small percentage of our customers” I’m always in that small percentage?

C’mon dreamies, get it together!

What do you guys think of blue host?

EVERYTHING still down 4:46est

back in business

Ok, It looks like we are back up and running. Thanks Brandon and DH! Tough stuff today.

Still down here. All of it. I am in a small, unimportant subset.

FATCOW are the worst!

March 23rd, 2012 at 1:51 pmMarcelo Oliveira Says:

became normal to have problems

DREAMHOST are the worst!

We’re back up. Thanks guys.

site “was” back for a couple minutes, but they’re down again

I hate to be a hater but these big outages are becoming quite frequent…

@Mana: I’m just saying that people tend to overreact to this kind of stuff. Your analogy is absurd.

albaldah ,theone , hanjin-shelf3 , plucker , hauler , milesaway , homiemail-sub4

small subset?

ya…

@Miguel: Yeah, I thought so. Thanks.

@DH: Thank you, guys! Our websites, webmails, sftp, and mysql are up again.

“Small subset of customers” … hmmm … “small” … yeah right ;-)

thanks dreamhost! You just made my decision to switch hosts super easy! I know my measly $400 might not mean much to you, but imagine if others follow suit…. Chapter 11 may be in your future.

peace out! Been with dreamhost for 6 years but all these debacles have caused my loyalty to fade away.

all my sites are down. (or at least I can get them to come up)
I can’t access my e-mail, dreamhost login or dreamhost in general.
Looks like it is time to shop for a new server.
perhaps 1and1.com
I only seem to be up 90% of the time not even close to what I expect from dreamhost.com

We have radio ads directing people to website for purchase of tickets. I like a credit cause this has happen twice before.

This is really terrible. We’re selling tickets online to our upcoming fundraising event, and people are giving up because they can’t get online to buy tickets, get contact or event info. We’ve been down multiple times in the last month.

everything down…….

My site was collecting donations for a starving african child, but because it was down for 2 hours the child died… thanks

March 23rd, 2012 at 1:56 pmdonna or steve Says:

our site is still down

websites online!,

still mails not working (pop and web)

It has been at least months since the previous BIG outage I noticed. But, they are common enough to make one wonder if maybe DH should be doing a better job and preventing them. At $25 for unlimited everything, pretty much, I guess I am getting more or less what I am paying for..

wtf???????’ DOWN SITES

Everything is down, the panel ad well !

I’ve been a customer since 1999, running a serious business, currently running a VPS and using DB and most email is google run. We do not have a super cheap package and pay a decent rate to host our dozens of wordpress backend sites. Today’s issue is related to DB as our sites that do not use a DB are still up. I normally don’t comment here, but today I felt compelled to do so.

I’d like to know if DreamHost’s CEO, Simon Anderson, is aware of how much of an increasing problem this is and that he really needs to address his customers honestly. Personally, we have seen a degradation in uptime and tech support service, especially over the last year. Being down for 1 hour to 10 hour stretches at a time is just AWFUL and as much pain as it is to make a switch, there are now other providers out there that do a much better job. Just look for cloud based vps or cloud based dedicated services and you’ll see a lot of choices.

We understand how much of a pain the migration will be, but our hand is now forced. Our pain threshold has been met. We’ve looked into several servies and will be moving over to a host whose datacenter is based in AZ, using their cloud service. Among other things, the tech support is a major improvement and the downtime is minutes (even during scheduled maintenance) not many unresponsive hours.

The cost? Slightly higher, but no longer unattainable. The base vps is $40 a month and we are adding some managed services and long term backups for peace of mind (something DH doesn’t offer), so that will bring up the price.

If y’all haven’t signed up for a free pingdom.com acct for your sites, I highly suggest you do so in order to measure your downtime with DH or anyone. They send you an email and text when the site is unresponsive and another when it is back up that catalogs the total down time.

Is Dreamhost now playing a different game? exit strategy? struggling? going public? Who knows, all I know is that its a change for the worse and they can’t seem to keep up. 300,000 is a lot of sites that can domino into too many problems when there are too few resources and I don’t know if the “happy dreamhost staff” is an accurate signature anymore.

horrible host. My site was hacked with common.php and .htaccess

Dreamhost does not offer a solution

This is happening more frequently!! Our website and emails have been down for hours and we are losing money by the minute!!!!!!!!!!! Come on Dreamhost, get it together.

So back near the end of last year, Dreamhost’s previously excellent uptime started to get a bit sporatic, so I got a monitoring service (I have about 35 non-profit and small business websites on the VPS).

In hopes of injecting some objectivity into the comments, here are my stats for the last 6 weeks:

WEEK UPTIME% FAILURES
mar 12 – 18 92.42 25
mar 5 – 11 98.21 6
feb 27 – mar 4 96.43 12
feb 20 – 26 99.07 3
feb 13 – 19 99.4 2
feb 6 – 12 98.51 5

So, my 6-week average is 97.34% – that equates to more than 26 hours of downtime! (and MANY hours of explaining to my customers and trying, less and less effectively, to defend Dreamhost as a good hosting option). Does anyone think that is acceptable? Does anyone else have similar stats to share?

Where are you migrating to?

Dear Dreamhost,

I can’t use you as a host anymore. You are too slow. There was a major disruption a few weeks ago, and you didn’t move me from that server, and the result is another major disruption. I can’t spend this much time fielding calls from confused or angry clients – and I shouldn’t have to.

I’ve been with Dreamhost for ten years, but it’s time to move on.

Our site’s been down for 24 hours (same sql problem as everyone here reports) – no reply from tech support, and now email’s down too.
Time to jump ship.

March 23rd, 2012 at 1:59 pmBarry From Eastenders Says:

What does smeg mean?

Our site can not even be reached now and e-mail is down. Looks like it’s that time of the year.

Ok it’s back…but customer gone :/ bad timing today…

=============N I G H T M A R E ============ H O ST ===========

AGAIN JABBER AND EMAILS ARE DOWN

NOW YOU GUYS MAKE IT YOUR HABBIT

SO PATHETHIC

SHAME ON YOU DREAMHOST.

PLEASE FOR GODS SAKE.

HIRE SOME EXPERTS SYSTEM ADMINS.

AND STOP REPLYING ON INTERNEES

AND STOP USING 1950s DATA CENTRES.

GROW UP.

How in the hell much lomger is this going to be down.
Have lost a full days business.
This is unacceptable!!!!!

Everythings here is down!!

March 23rd, 2012 at 1:59 pmBarry From Eastenders Says:

@Pissed I’m migrating to your mother

Everything worked for a whole 5 seconds. And now its back down again. Good job guys!

Site came back up, but was hanging on script to display ads and PHP code that has worked for months. Now the site is down again. Going on hour # 3 of this. Yikes.

This is taking too long!!!!!!!

March 23rd, 2012 at 2:00 pmBarry From Eastenders Says:

@neo Follow the rabbit…. out

gone again…

Report from small subset: Web sites up, email still down

Also looks like dreamhost.com is down too. A+ for effort.

March 23rd, 2012 at 2:01 pmBarry From Eastenders Says:

I’m going to Disney Land!

hmm… hopefully this is easier to read…

WEEK UPTIME% FAILURES
==================================
mar 12 – 18 92.42 25
mar 5 – 11 98.21 6
feb 27 – mar 4 96.43 12
feb 20 – 26 99.07 3
feb 13 – 19 99.4 2
feb 6 – 12 98.51 5

6 week average: 97.34% equals roughly 26 hours of downtime

BACK DOWN, UP FOR 3 MINUTES!!!!!!!!!!!!!!!!!!!

WOOWWWWWWWWWWWWWWW DREAMHOST WHAT THE FUCK

Anyone who has a mission critical site that is down right now should have a backup server. Nobody runs mission critical services off a single server and expects it to be up 100%. Redundancy and backup servers exist for a reason and any IT/web professional should understand this. Get a grip people and stop being cheap. Get a backup server and quit crying

Thanks DH, now this gives me ammo to my boss to tell him its time to find a different hosting site, keep up this fantastic service and I bet other hosting sites will give you a cookie.

Mail has been down since 11 am!!!
This is very bad for business.

Am I correct that this is more than just an internal server issue? The status leads me to think there may be another issue they are unaware of – ie: all of our sites are down. Come on DH! You can fix this! I believe in you.

Pretty much all my sites are down. This is unfortunate because Friday is a popular day for my sites. Hopefully Dreamhost can get this resolved soon.

As easy as it would be for me to bitch about how much money I’m losing, I’m not losing any money as the sites affected are all non-profit. However I am losing faith, I have been a DH customer for a long time and outages used to be a rare occurrence. In fact, that was what had me pulling sites across to dreamhost in the first place. I am to move all my sites away in the coming weeks even though the destination datacentre is known for outages, because it still has fewer and shorter outages.

Yup, my site just went down. Didn’t go down two hours ago like the status indicated, just in the last 10 minutes. Web Panel is down too. At least my 500 error page works. MyPHPAdmin up/down sporadically.

Service was up about 15 minutes ago and now everything is down again.

I take it back – we’re down again now too.

I was up for a minute there! Now I am gone again.

We were up for 5 minutes. Now we’re down again. This is torture. What is going on that we would be up for 5 minutes, then down again?

Between this, the recent outage with stampeders and the subsequent downtime (not to mention the outrageous explanation that somehow our own sites are to blame for script usage in a basic wordpress install), I’m moving hosts. This is unbelievable.

I wasn’t experiencing any issues until about an hour ago. Started with one user’s email, now it’s all users email, FTP, website… It’s just getting worse and worse

Have been experiencing issues with regular email as well. Thank you in general for your service; am overall very very pleased with Dreamhost.

Wow…site is back online now, and seems snappier than ever. You should leave it “re-routed” because it’s way faster than before…

I’m with everyone else here. I used to sing the praises of DH as the best hosting company in the world. “They have problems like everyone else, but they are relatively open about them and resolve them quickly.” That is still mostly true, but you now experience major outages far more frequently than any other hosting solution I’ve ever know which completely nullifies the fact aforementioned statements. I find it harder and harder to recommend DreamHost to people now. Like others have said, you need some automatic redundancy built in to your network, so you don’t have to dispatch a team to reroute traffic.

With that said, good job getting things back online quickly today. I was worried this was going to be like the last issue that seems to linger for days for many customers…

Hey my friends…, Very saddly writing this stuff.. I’m about to get fired because of this issue…

We are seriously considering moving our hosting to another company. Our email has been down far too often, and now our websites are down as well. Not impressed at all.

Update Mar 23rd, 2:00pm PST: We fucked up everything! Now all is down! panel, dreamhost.com and everyone’s else shit! Glad our blog is not hosted with us! — Brandon

Up, Down, Up, Down, Up, Down. One step ahead, two backwards… :(

WTF still down!!!

you’re really ruining my day!

My site came back – and now it’s down again >_<

bunda still down, and can’t connect to support to submit a ticket

Boo hoo been without email for hours. This is getting ridiculous. :-(

and now it’s back down…guess I spoke too soon…

http://panel.dreamhost.com/

DOWN

EMAILS ARE DOWN

JABBER SERVER DOWN

FTP ARE DOWN.

WHOLE OF DH DOWN.

WHY YOU SUCK THAT MUCH.

WHY YOU DONT START DELIVERING PIZZAS INSTEAD OF DOING FANCY BUISNESS OF SERVER ADMINISTRTION.

NOW STOP PLAYING AROUND.

EITHER SHUT DOWN THE COMPANY OR RUN IT LIKE PROFESSIONALS.

HOSTING IS SERIOUS BUISNESS…

ITS NOT A JOKE….

YOU GUYS MADE IT LIKE A JOKE….

IM FUCKING SICK OF YOUR DAILY ONGOING DOWNTIMES…

i’ve been a DH customer since 1999 as well and this is the worst outage I’ve personally experienced, probably exacerbated by the fact that I had time-critical applications going just before the outage started. No amount of apologizing from DH is going to help with the impact (financial, time, confidence) this has had. At the end of the day, I probably wont switch to another host provider cuz of the hassle, but will certainly be researching around for alternatives. A “we’re real sorry” just isn’t going to cut it this time.

March 23rd, 2012 at 2:05 pmLongtime DH Customer Says:

WTF!!!!! Gheeeeeez……. FIRE whomever has not fired whomever needed to be fired after the last outage!! I can not believe you guys have totally dropped the ball these past few weeks. Years and Years of success can so easily be swept away with problems that can not be explained nor fixed.

Man this sucks and seems to be happening more and more all the time.

Can anyone even connect to their admin panel? I have been getting this:

The connection has timed out
The server at panel.dreamhost.com is taking too long to respond.

Everythign back down again.

Oh, good. No more “Error establishing a database connection” … Now my servers are totally unreachable.

Down again. Dreamhost.com is not even up

HELLO? All my sites went down….Dreamhost.com is down……where is the update????

Always tends to happen when my accounts are sending in their forms for the 2:00 deadline. I fell punished by DH.

Come on guys… This is how I make my living. If you can’t sort out your issues I’m going to have to move my customers to another host.

I mean, I can’t even hit Dreamhost.com right now. That’s certainly no “small subset”.

It figures… All my clients’ sites are down now too. Looks like it really is a cascade failure involving more than homie cluster. Bummer.

Amazing 4 hours Down, What kind of people can stand this failed, i will lose some clients of course money. im sure.

Been with DH since 1999, and this is actually my first down time yet.

That said, it is happening when my portfolio is in the hands of my largest target companies. Ugh, screwed.

Hi,

All websites in my account are DOWN,

Please fix the issue quickly!!!

Another very unhappy customer that is HIGHLY considering moving our site to another host provider. What is going on?! These continuous outages are ridiculous!

Pooooor Brandon! ;)
Even the Panel is down from Belgium.
May the Force be with you guys! :)

So… my sites been down for hours now… If i ask nicely would you please put it (and everyone else’s back up?)

Site is down, can’t access panel.dreamhost.com or dreamhost.com itself.

I’ve only had all my DH clients go down once before. That was a long time ago.

You know, normally downtime affects me little. But this email outage is a serious problem for me right now this minute. Could not have happened at a worse time. Dead in the water for several reasons and all tied up with email. Speed is of the essence and I have nothing at all. And all my websites down and dead too.

Sucks to be me, huh?

Redundancy and failover – look into it, eh DH?

are the DNS still down? i can’t see my site that has only the domain registered at Dreamhost…

what the hell is happen???

Today was also a big day for client demos myself. I just lost up to 10k in business. Thanks DH.

I guess I’m joining the the bandwagon now because ALL of my domains are down since 5 minutes ago as I was working one of them. What the heck? Do you know that this is costing me precious time and money?!

Move the power bar that the servers are plugged into so the janitor stops hitting the power button with his broom. Mount it on a wall or something.

Okay this is getting pretty ridiculous. You guys know how to configure BGP, right?

All eight sites of mine are down. Hmmm… Saturday morning here in Sydney – a busy period for me. Any more news fellas?

Define “small”?

This is happening with way too much regularity. Very disconcerting. Please get your act together Dreamhost. Take a deep breath Josh. People understand.

I just want to say, for all of you complaining about your email being down or your time-sensitive blah blah…

If it is time sensivite/secure/important, it should be done in more than one place.

For your email, you should be hosting it with gmail anyway…

I’ve been gone since 1:30pm EST and just got back to my computer to find none of my email accounts connecting.

Folks! We can’t even log into the Dreamhost dashboard. The site is completely off line.

Get a handle on this quickly. We cannot afford this happening agian and again.

March 23rd, 2012 at 2:08 pmBarry From Eastenders Says:

The devil is horny

Im very sad to say this…
But at these days, dreamhost is unable to handle the big amount of their customers.

Well everything got an end, only a sausage got two.

Networking 101 – check the uplinks, first. Maybe change everything to DHStatus host! They seem reliable.

Yow. Yes, something just went very pear-shaped. :(

I think Dreamhost puts this comment post here on purpose so they don’t have to listen/read to us whine.

i don’t have emails!!!!!!!!!!!!!!!!!!

ARE YOU FUCKING KIDDING ME

Everything is down…. website, panel, everything!

Sorry about all this guys. Turns out that the CD tray isn’t actually a cheese tray like I thought. There’s goo all over the servers. And I don’t think its ever going to come off with out moving to Operation X. That’s right, I’m going to release Mice into the servers to eat the cheese. There’s an 85% chance that mouse pellets may cause lingering problems in the future, but that’s better than the 100% chance of failure you have now. Thanks for understanding that this is only affecting a small, I mean all, I mean… Thanks, — Brandon

March 23rd, 2012 at 2:09 pmBarry From Eastenders Says:

@Mick really? You need someone to define small for you?

My site won’t load. Any news here please?

What’s going on over there? You had it back for like a millisecond. And now, all sites are down again. Please fix, or give some sort of update.

Dreamhost is completely down now.

Things just went from Bad to Worse.

Is there going to be a week that we DON’T have an outage?

Having a really hard time getting work done this week…..

All my domains and mail went down around 10 minutes ago

502 Bad Gateway

nginx/0.8.53

hay sizin datacenterinizi amq

all my sites are down, https://panel.dreamhost.com/ also down.
im sweating blood here. Pleas please please, be quick.

March 23rd, 2012 at 2:10 pmBarry From Eastenders Says:

@Brandon FTW

Nice job guys

All my sites just went down. Panel is down too!

I was just about to sign up for another DH account for a client but I think I’m moving them over to HostGator or anywhere else. Who’s with me?

c’mon this is my busy time!!

… only effecting a small subset….

Luiz Jr. (Blog Boa Viagem), I hope you dont get fired.

I was in the middle of a presentation to a new client when all this shit happened.

I`m thinking about move to Amazon….

site still down, depending on databases, mail also affected :(

No e-mail and cannot access our company’s web site at 17:06 EDT on Friday. This is one of the more serious outages.

This sucks, after being with dh for 5 years, I hate to have to move, but this is nuts.

All of my customer sites are down – again. Heard from several of them already saying they are changing hosts :(

Can’t even get to dreamhost.com (getting 502 Bad Gateway error). I honestly don’t know what’s been going on lately, but these interruptions in service are quickly becoming very annoying…. as I was typing it, Dreamhost website is now accessible. My sites are not…

502 Bad Gateway
nginx/0.8.53

March 23rd, 2012 at 2:11 pmChris Schaffner Says:

Proposal due now and email is down

Yet again, let down. Can’t get to panel, can’t get to dreamhost, can’t get to any of my sites, yet told I’m a “small section.” Last time you all cost me 7 days of data that went POOF because you lost it when you had to do a data restore. Fortunately, I don’t rely on shared hosting to make money. This keeps up, however, and I’ll not be using up for hosting that DOESN’T make money. Too many times since the beginning of the year. Stop with the stupid newsletters, and FIX the issues.

Will someone please meet me in the bath house?

It’s lonely with my websites down and I just need a shoulder to cry on…

My websites are all down… heeeeelp!

I’m moving to mediatemple. I feel like George Bailey’s customers going to Potter, but this is how it is. Lost an entire day of productivity.

My site and the dreamhost panel are down, so I can’t even submit a ticket to support. These outages are really getting aggravating. They always seem to occur during periods of what would otherwise be high traffic for my site.

March 23rd, 2012 at 2:11 pmBarry From Eastenders Says:

@David this is my BIZZAYY time!!!

Well at least they host their own website on the same network, so they are also down when we are.

Looks like their mail is also hosted on the homie cluster:
dreamhost.com IN MX
preference: 0
exchange: mx1.sub5.homie.mail.dreamhost.com
14400s (04:00:00)

Wonder if they even see support request when an outage like this happens?

Back down :(

Great! Few minutes ago, I was thanking DH for the live websites again. Now, it’s getting worse, even the panel is certainly down so as all the websites hosted by them, webmails, sftp and mysql. (sigh)

I totally have a potential employer looking at my work today too – earlier, my email was just down. Now EVERYTHING is down.

DrEaMhOsT you should host yourself on godady =)

March 23rd, 2012 at 2:12 pmBarry From Eastenders Says:

@Random Emosexual I’m all over that action sweet cheeks, lubing up and I’ll be right over!

Small subset? As if that makes it less of a problem. Each customer is a paying customer. 656 disgruntled replies and counting…

come on guys, what the hell is going on

My site has been down for a few hours now. I’ve been having huge traffic from reddit all day. As an entertainment website and all the coverage on The Hunger Games – this is losing my site significant income from ads.

@Craig – You don’t consider your front page your store front, your visitors your clients, and the ability to run said website allowing you access to your register? Because that seems fairly apt to me.

is out of service

What is going on? Please, inform.

Mail down in Berlin…wanting to go throw things…

This morning I made joke about DH being wacky lately.

I should never make likes like that.
3 VPS instances.
1 Shared hosting account.
Can’t log in to panel.
Can’t connect with FTP.

This is really messed up.
I have recommended DH to many people.
Never again.
They all look at me now like I am Hitler.
I do not need this.
There are better alternatives out there than this.

No data received
Unable to load the webpage because the server sent no data.
Here are some suggestions:
Reload this webpage later.
Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.

Down Down DOWN … Webmail … Websites … Even DREAMHOST.COM ….
Why does this server company have this amazing ability to go down
while others constantly enjoy a good uptime?

That’s it. I’m moving. Been to much shit the last few months.

In trying to “fix” this issue, they just made it worse. My site was up and running just fine until a few minutes ago. Now I can’t even bring up dreamhost.com. I just got a 502 Bad Gateway page.

our website is down! we were responsible for the crowdfunding portion of the JOBS Act that passed the US Senate yesterday! PLEASE GET US UP ASAP!!!

March 23rd, 2012 at 2:14 pmBarry From Eastenders Says:

@Daisy Totally? Really? Totally was the word you wanted to use there? Rather than just, you know, not putting an additional/pointless word in?

Why Oh why are we always part of the “small group” impacted. And always during a promotion. Always. Always. Always

All my sites are down. I don’t find the humor in Brandon’s statement “that this is only affecting a small, I mean all, I mean…” I’d rather have a clear update that says either I’m just among the small group, or I’m among the all.

Yawwwn. Another outage at Dreamhost. I say we vote for something unique…like uptime maybe.

I’m not one for whining, but I’ve been complaining for months after years of good (not great) service, and all I hear is that they can’t see that my service is going down and it’s up to me to diagnose the problems. So it’s time for a new host, but I don’t want to move from bad to worse. Does anyone know a good reliable hosting service for a dozen or so wordpress blogs, only one if which is really important? I’m paying for VPS now, which means I’m spending a fair amount of money for unreliable service. I’d be comfortable with $100/month if it wasn’t ever breaking.

All my sites are still down as well… can’t connect to the Tashkent server via SSH or FTP. The Panel doesn’t work either… this is fun!

Also this sentence above your comments is wicked: “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.” well… I can’t get into the panel so what is the best way to contact Dreamhost?

I detected that my VPS made some outgoing TCP connections from some daemons. But then it went down again.

99.76% uptime so far in March. Pretty crappy guys…

Oops! Google Chrome could not connect to dreamhost.com
Try reloading: dreamhost.­com
Additional suggestions:
Access a cached copy of dreamhost.­com
Go to dreamhost.­co
Search on Google:

Not only is my site down, but I can’t access the Dreamhost site. Your post seems to be sugarcoating your real problems at the moment.

March 23rd, 2012 at 2:14 pmBarry From Eastenders Says:

@Wodie Try Viagra.com, gets me up every time

Ooops, really pissing-off outage going on right now!

This just in the mice have all died. I repeat the mice have all died. The servers are now covered in both cheese and blood. I don’t think there’s a chance this will ever come back up. My sincere apologies to those in the smallest percentage possible of you who have felt sympathy for me. The rest of you can rot in hell. Sincerely –Brandon

@Barry From Eastenders lolololololololo

seriously? again?

“This issue is still only affecting a small subset of our customers…”

This is happening every single day…when is this going to stop being so frequent?

I’m sorry. It’s down because I paid my bill this morning. It seems that every time I pay my DH bill, I can count on a MAJOR downtime within hours.

I have to admit, I am getting burned out over the downtime at DreamHost. Unlike some of the other people with their hair on fire here, I am not losing “thousands” of dollars. What I am losing is my confidence with DreamHost.

I have been actively looking at Spry.com lately, basically for it’s cost and reliability. What I am finding is that what I am giving up for “unlimited disk space” I am sacrificing in resources and reliability. Because I store over 6 thousand mp3 and flv files on DreamHost, I always thought I needed something with unlimited space available. Now I am looking at a different way to store and access these files and still run a decent website.

Basically, I think I may have outgrown Dreamhost.

Dreamhost ! you should pull out of the world hosting market, try your chance on Iraq or Syria. goverments don’t need to block every suspicious web sites thanks to u , beacuse u already done it.

March 23rd, 2012 at 2:16 pmBarry From Eastenders Says:

@Brandon dude, I’m sorry about the mice :(

Well, if it’s a routing issue, it’s likely to get worse before it gets better, but I’d’ve thought it should have been better by now.

Routing problems are *ugly* – give me a crashed disk any day.

ill need a truck full loaded of vaseline for my, untill now, virgin ass.

Site is down again.

Can’t get to the Dreamhost panel either to report the outage.

Sorry I crashed the server guys, I was trying to upload and serve 10 petabytes of underage monkey porn. Now I know that DH is not the place to do this.

Honestly, the comments here are pretty funny.

What is extremely funny is the fact that even dreamhost.com is down at this point. Still don’t bother sending emails, they will turn everything against you anyway, it is usually customer’s fault as I noticed lately in my dealings with DH support. This year was the worst but then again everything works flawlessly when you just pay for the service and you don’t use it, if you get my drift. Probably their policy is based on quantity and not quality, get enough subscribers so if some of them leave we still have enough to get our fat paychecks at least. I say it is time to get those professionals on board if you guys still want to be in business 5 years from now, one outage a week is getting ridiculous.

Oh and how are we supposed to open a ticket at this point? Have you guys considered hosting dreamhost.com let’s say, at Amazon?

This is absolutely unacceptable.

Please add all my sites and customers, webmail, email, ftp, and web pages to the list of DISGUSTED customers. FIRE YOUR NEW CEO AND BRING IN A PRO TO SET UP SYSTEMS THAT PREVENT THIS ONGOING SLOW MOTION DISASTER! Your service was good for many years before the current management.

WHAT THE FUCK! ALMOST 30 HOURS OF DOWNTIME IN THE PAST MONTH! YOU HAVE COST US LARGE AMOUNT OF REVENUE AND PISSED OFF CUSTOMERS!

@Barry From Eastenders Totally.

Dreamhost, this is pretty bad. The last time there was a huge outage like this was just earlier this month:
http://www.dreamhoststatus.com/2012/03/05/connectivity-issues-to-some-of-our-datacenters/

March 23rd, 2012 at 2:18 pmBarry From Eastenders Says:

@Pissed Off LOUD NOISES!!!!!!!!!!!!!!!

My Site / Mail down. (and while we are testing alpha)
My partners site /Mail down.

Cannot access https://panel.dreamhost.com .

-.-

I have 4 websites with DH, ALL OF THEM ARE DOWN!

You guys… wth come on.

I have been reading some reviews, and it looks like JustHost.com is near the top of every list for host providers. Anyone have any experience with them?

March 23rd, 2012 at 2:19 pmBarry From Eastenders Says:

@Daisy Brilliant

My site is down as well.. and so is the dreamhost site..
I assume they’re working on it.. this is actually the first problem I’ve had..

Great. Guess I won’t be finishing that website before 6pm so I can send to client for review. Thanks Dreamhost. I feel the love. :(

Hmm, I would like to make a support ticket, oh wait, ha, yeah there entire is site is down.. a company that cannot even keep its own site up?!!

We’re just over 100 hours of accumulated downtime in the past 8 days. This is fucking disgusting!

Yes, we are in “hell”

Yay! Thanks for the detailed updates.

A small subset? Not only are thousands of customers (including me) affected, but I can’t even get on DreamHost.com

It’s an absolute pain in the ass to switch host, but DH is forcing my hand. :/

“small subset” my ass

Just finished rebuilding my three hacked (because of a DH mistake) sites, and now another outage. As soon as possible, I will be moving my sites. An occasional outage us understandable, but continual outages as well as security issues – not acceptable. Another 8+ year customer gone.

F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!! F-YOU!!!

March 23rd, 2012 at 2:20 pmBarry From Eastenders Says:

@Derek JustHost.com raped my goldfish after my cat ate him…. I don’t know how they did that? I’m angry but still kinda impressed at the skill involved… either way, THEY BE EVIL DOERS!

Mail, webmail, and website are all down. Also, can’t even get into the web panel.

@Barry Thank you for your condolences. Those mice are really my only friends. Dream host keeps me locked up in the basement with access to very little other than twitter. I found those mice 2 years ago in a corner and we had raised a significant little family together. One of them died recently and we shut down the servers out of respect a few weeks ago. It is such a tragedy now that the rest of the family has died from those very same servers. I will forever morn my little buddies. The rest of you need to think long and hard about what you have. I only have one friend now and his name is Barry. Chew on that while you wait for your “websites” to come back on line. –Brandon

Any updates? Just talk to us…

dreamhost got problems

Only a small set of users are effected? Then why is Dreamhost.com down?

website, email, squirrel mail all still down, can access control panel. it’s friday afternoon and the staff just want to go home and not wait for email to go thru to a client. and of course, helpfully, they are letting me know about it.

not just mail, webmail and mysql. www is affected as well.
it caused all of my websites to be offline as well.

Obelix seems affected now…

dougzilla Says:
“I have recommended DH to many people.
Never again.
They all look at me now like I am Hitler.”

Umm, yeah. That’s a bit much, don’t you think? Or you need to meet a MUCH better class of people, either way,

COME ON!

I would like to take advantage of the uptime guarantee by posting a support ticket, but the panel is down. Hopefully this comment will be proof that I am am aware of the problem.

-Sharp

A hosting company should go bankrupt before letting something like this happen.

I think I’ll start looking for another host today.

It keeps getting harder and harder to tell my boss that dreamhost is in line with industry standard uptimes…. c’mon guys.

well FUCK

I moved all my stuff to a dedicated server ($99/Month!) 3 days ago to get more stability and stop experiencing this kind of unpleasant episodes…

So it seems that money can’t buy everything!

Do they even read their comments on their posts anymore?

websites down again.

Yea what Solomon said… Everything is down

I’ve been with Dreamhost since about 2005. In that time I hardly ever had any downtime at all, life was good.

In the past 3 months my sites have been down about 5 times including now, FTP went down, emails went down, had to change all my passwords because of that major security breach.

I’m really trying to be patient here but, if this continues much longer I don’t think I have a choice but to move elsewhere. Shocking amount of downtime.

If this is your datacentres fault and they’re run by someone else, my advice Dreamhost? switch server companys NOW, cause they seem to be going down the toilet.

This is totally ridiculous!!! We’ve been down for almost 3 hours and still nothing. Can’t do anything, losing tons of money in productivity. You guys suck, we’re going to be changing.
This has happened WAAAAAAAAAAAAY too many times recently.

So, uh, who hosts THIS blog? It seems to be up ALL THE TIME.

My sites have worked periodically throughout the day, but I haven’t been able to receive or send mail all day. Right now, I can’t even connect to the Dreamhost site itself.

My sites were fine all day, then just now are completely down along with panel.dreamhost.com. I wonder if this is a new issues since they were working at noon PT.

Come on guys! My database access is not working!

LAX is down!!? LoL?
Flood?

I’ve been a loyal customer for years and brought several people to Dreamhost. Lately I’m regretting that. Every person I’ve brought to Dreamhost is losing money when hosting goes down. And it has been happening too frequently. What happened to “99.9% uptime”? These recent outages are shaking my faith in Dreamhost. I need more than apologies and empty reassurances. I need a reason to stay.

Been down since 12 CST..this is BULLSHIT..who do we get to pay for our days expenses, employee’s, etc.

lol @ dougzilla. I’ve referred thousands of dollars in referrals to DH from my clients, now I feel somewhat responsible for their website being down. Go figure.

pork pie: eheheh nice one

“We have redundant routes setup to prevent this type of effect when a provider issue such as this one occurs, however the routing tables in question did not self correct…”

haha, isn’t that how Titanic sank?

unacceptable!

Put another chimp on the job and get this shit working!

This is the last downage I’m sticking around for.

Time to find a new host.

The dreamhoststatus is slow now, it will be down soon !

Where is dreamhoststatus.com hosted? It’s the only thing working fine today!

March 23rd, 2012 at 2:25 pmGregory McArthur Says:

Go dammit, I’m losing ad revenue! I need those pennies for my ramen noodle dinner.

You guys are really starting to look like amateurs, I don’t know what you guys have done in the last year, but there must be retards over there, because every month there is some huge issue that has sites down. I have been with Dreamhost over 3 years and in the last 4 months it has been a damn night-mare. My sites are ranking in Google and now you are down. I hope you guys are reading this, because eventually you will loose all your clients for not being able to deliver.

Check out Tidal Media in Portsmouth NH, they’re reliable: http://www.tidalmediagroup.com/web-hosting.cfm

OK… so we’ve been down since you guys started correcting the routing tables. What gives? Maybe they need to be re-re-corrected???

The announcement and updates say ‘a small segment of our customers’ but the Dreamhost site, the blog, the forum, all down for a bit now, with this error:

Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.

Sounds to me more like all our customers and potential customers and all their customers and potential customers.

Sigh…

Down Down Down Feel Like The Club Is Burning Down
Down Down Down Down
Down Down Down Down
Down Down Down Feel Like The Club Is Burning Down
Down Down Down Down
Down Down Down Down

I’ve been with Dreamhost for something like 10 years. I don’t ever remember things being as messed up as they are lately. Is this what happens when a bunch of investors take over a great company? I’m getting the strong feeling my beloved Dreamhost is just going to sink further. It would be nice if the investors would get off their bums and make some effort here.

Third outage costing us business in 3 months.

I’ve already been lectured enough how it’s my own fault if I choose Dreamhost.

So… I guess I’m just chiming in that this is getting old.

Been here ten years now… Lately it is bad…taking steps finally to wean ourselves of the DH..

Wow. WTH has happened to DH?
This kind of deja vu I don’t need.

@ at least i got (brightly) all mt mails hosted on gmail.
THANKS GOD, you all should try same.

Wow. What can I say? This is just unbelievable.

Pull it together DH – 2 outages per month is taking the piss

I wish everyone would stop bitching. Do you not realize that shit just happens? It’s not like someone at DreamHost said “Ha I’m gonna pull this plug to piss off hundreds of our customers”. Just shut up and let them fix what’s broken, you bitching isn’t gonna make it go any faster and you just make yourself look like an asshole. Also, no one cares if you’re switching hosts…just shut up and do it.

my site this is the height callus

?? any updates? i can even get to the dh site or any of my sites

ITS A NIGHTMARE…………….

I HAVE 3 ACCOUNTS OF DIFFERENT DH SERVERS….

HOSTED ALMOST 50 SITES OF MY CLEINTS….

ALL ARE DOWN…

WTF IS GOING ON….

IS THERE IS WAY TO SUE THEM ???

FOR MESSING UP WITH THIER CLIENTS….
THEY SHOULD BRING TO JUSTICE…

YOU GUYS SUCKS

YOU MUST BURN IN HELL…

When we have an issue like this, what is the way to make a claim?

If me or my client lose money with that there are any way to obtain a money return from DH,

I demand to DH team to get me an answer about this money lost,

Regards,

Juan

Darn… It’s Friday and I want to quit for the day. I cannot upload my files with SFTP. Just got error message “I/O Error; Connection failed. The kexTimeout (60000 ms) expired.”

I also can’t download our email. I have an important message to send to a client once I can update these files.

Oh, I can’t access the admin panel (https://panel.dreamhost.com/) either. ERROR: “Safari can’t open the page “https://panel.dreamhost.com/” because the server where this page is located isn’t responding.”

Can’t submit a support ticket.

Can’t view any of my websites.

Maybe DreamHost can give us a bit of compensation for all of this downtime?

Doesn’t hurt to ask :)

I just had a client call and he cant access our website. Makes us look very bad…

GOOD LUCK

March 23rd, 2012 at 2:26 pmDonald Trump Says:

YOU ARE FIRED!

Have an eye on the detail Just below the LEAVE REPLY title..

quote “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

:S

this is ridiculous. everything is down. i have an app on the iPad where it needs access to these servers for customers to download. this is happened way too many times recently.

Need more power on Dreamhost? Buy some powerfull servers and please increase the limit of RAM to 120MB for shared hosting. Thank you in advance. Long Life Dreamhost!

@HaLo2Fr – So you’re bitching about people bitching?

i just spent this quality down time looking at godaddy as a new host. it’s going to take a pile of work to transfer my stuff but i think it’s going to be worth it in the long run. sorry dh, i tried for years but an email outage for multi-hours on a close of a work week means i’m going to be looking at a pile of wtf emails from clients. ugh. stress. it’s my own darn fault for staying so long.

I wonder if they’ve managed to cut themselves off so they *can’t* update the status.

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU

I usually have patience for this but this is a bit much. I can’t even open the panel.dreamhost.com to send a ticket to complain about this. This is exactly what happened the last time they went down, first the email, then the websites, and then their admin panel. I need a drink to ease the endless barrage of pissed clients on this server.

March 23rd, 2012 at 2:27 pmEder Acevedo Says:

This is absolutely unacceptable.

again! … nooooooooooooooooo!

Dreamhost sucks!

I have a new provider: I want the service again for the migration.

Thanks.

“Bad Request

Your browser sent a request that this server could not understand.”

Trololol!

Good to know that even the least experienced network engineers can find work these days.

March 23rd, 2012 at 2:27 pmEverything is Brilliant Says:

Barry – you must be British or something? I’ve heard British people hate the word ‘totally.’

Can’t get to dreamhost.com, web panel, or our website. This is getting a little old, guys. Seriously. I don’t know how your website can be down and you say it’s only affecting a small subset. Maybe you don’t realize the scope of whatever problem you’re having.

down again?!! I wonder what my uptime percentage is at this point this year.

March 23rd, 2012 at 2:27 pmBarry From Eastenders Says:

@neo

I

LOVE YOU

THE WAY YOU TYPE YOUR COMMENTS IS SO

UNIQUE

AND YOU HAVE A PURTY MOUTH X

Another outage?? Sigh….

Maybe we should email Dreamhost and tell them the issues we’re having…oh wait….EMAIL IS DOWN!!!!

Have an eye on the detail Just below the LEAVE REPLY title..

quote “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

:S

WHAT IS ALTERNATE SOLUTION FOR CHOOSING HOSTING.

DO SOMEONE KNOW GOOD HOSTING COMPNAY

PLEASE LET ME KNOW.

my sites and mail that were unaffected at 3pm are now completely down – along with my sites that were down to begin with.

NOT moving in the right direction here DH.

@HaLo2FrEeEl – Have you ever seen the movie Airplane? That guy pulls the plug and the landing lights go out… I think he works at DH.

I can sense CLASS ACTION in the air ;)

we need a solution!!!

Is dreamhost still down? I can’t reach my sites nor shell.

I moved my wife’s blog to DH and now she’s mad at me.

You have no idea how much grief you’ve caused me.

I’ve been recommending Dreamhost to others for nearly a decade now. I love you guys, and I’m hopeful that you’re taking steps to fix the problem. Please don’t let the faithful down.

March 23rd, 2012 at 2:29 pmBarry From Eastenders Says:

@Everything Close, I am made from Cheese, it’s a kind of meat, a tasty yellow beef… I milk it from my teet and try to be discrete

@Donald Trump
Who are fired? There are only machines!!! LOL, did you see Terminator Salvation :)) ?
They are NOT humans… ONLY machines :P

How can the Dreamhost.Com site be out. When this is resolved, I am moving to a different provider. You guys fucking cannot do what you charge. It’s just I am sorry…..I am sorry…………DO SOMETHING TO CORRECT THE ERRORS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I am defiantly leaving!

From Kevin Wohler “What happened to “99.9% uptime”? These recent outages are shaking my faith in Dreamhost. I need more than apologies and empty reassurances. I need a reason to stay.”

I would have to agree, simple apologies for downtime and security breaches are no longer enough. Stand up to your guarantees DH and start refunding money to people… to keep from a class action suit. To keep people with your service… no suggestions on my part, but it sure looks like you’re going to lose a large majority of the people on this page!

I’ve started a petition here to demand a public response explaining just what is going on and what they’re doing about it.

Sign on and let’s get some clarity from the service that we trust our sites with: http://www.ipetitions.com/petition/fed-up-with-dreamhost/

Hard to believe it’s a small “subset” of users when there are this many comments… just saying.

March 23rd, 2012 at 2:30 pmBarry From Eastenders Says:

@Doug New Wife required methinks!

come on, 30 min isn’t update

Hay que fastidiarse OTRA VEZ!
bye bye Dreamhost. No more contracts from me to you.
It is absolutely inacceptable. All my services are down, and still waiting for a solution.

This is getting worse today instead of better! I was up and updating my site until a few minutes ago. Your panel site is even down! We certainly aren’t being told the whole story.

March 23rd, 2012 at 2:30 pmBarry From Eastenders Says:

I just used the word “methinks” YAY!

Seems like there’s been a lot of downtime lately. I don’t think that is a stretch or exaggeration.

“This issue is still only affecting a small subset of our customers”

now over 800 users.

My clients ares a reaming the death of me…

ALL DOWN =/

Hello, my websites are still down, please fix the problem.

March 23rd, 2012 at 2:31 pmMike Villani Says:

Completely ridiculous. Been a member forever and have never had issues like we have seen over the past year. Definitely going to drop this piece of junk ASAP!

I will shortly be replacing all of your websites with this to allow your visitors to know there’s a problem! Thanks –Brandon

maldito dreamhost

We’re new to Dreamhost. I was bragging about DH but a couple of days ago I heard some serious DH complaints. Now our sites down during a critical communication time with our audience. This is really painful.

oh goody! The day I set aside to work on the websites I manage is the one day the datacenter explodes, affecting all sites. Why did I bother getting up this morning? Blargh

Once upon a time, there was a hosting provider.

It was doing fine until one day, it got hacked. Then it moved all its VPS machines to cheaper servers that started throwing off errors. Then it went down because of the data center.

The results were manifold:

* Several users committed suicide.
* Many clients fled, and most users starved.
* Several users were killed by spouses or coworkers.
* A giant class action law suit commenced.
* The rest of the users left.
* NORAD, which was hosted on the hosting provider, crashed and started a nuclear war.
* Everyone died.

As you can see, this is a life-and-death situation.

Ask Dreamhost for your downtime refund and realize that getting a new CEO means they’re cutting costs. Your VPS machines will be slower. Your bandwidth will be degraded. Don’t expect freebies you’re not grandfathered in for. This is how corporations grow up.

I’m Jewish….I’m going to go pray that service is restored by the time the Sabbath is over….

DREAMHOST YOU SUCK…

I lost an incredible amount of business and was unable to test a new site….

Enough is enough….

BTW….what happens to the email we don’t receive?

Fix those iptables!!!!!!!!!!! Please,please,please.

Barry From Eastenders Says:
@

ARE YOU SOME BITCH OF DH……………………………..AND EVERYONE WHO SAYING STOP FUCKING DH

This is really bad!!! We cannot access email or the website. We need this fixed ASAP!!!! We can’t work like this!

clearly by ‘small’, the intention is to minimize the apparent impact on their customer base. The usage of the adjective has had a completely opposite effect on those directly impacted, however. I just want to wring DH’s necks right now.

We’re down too. This is happening far too frequently. Not happy.

I know it’s just a ‘small subset’ but when it’s YOUR domain, this really sucks. Please fix. And please let us know somehow.

aaaaaaand now my account on Stampeders has no mail function either

My site was back up and running and now it’s DOWN AGAIN.

This is INTOLERABLE.

HOW is Dreamhost going to make up the loss of this day to me?

all down for 2 hours now… :(

Still down and out — it is 5:30 pm ET — please advise when you will be back on line.

Wow. I WAS having a bad day, but not nearly as bad at the day DH is now having. :P

GUYS NEWS FOR YOU

Duplicate comment detected; it looks as though you’ve already said that!

ITS A WORDPRESS SITE THAT LEAST DH CAN DO FOR US TO POST COMMENTS.

mail has been down for me for several hours.

“This should only be affecting a small subset ”

That may make you feel better but it sucks for us and is bloody insensitive !!

hey….this is the 2nd time in 4 weeks that I have been unable to access, edit, or finish my websites. do you really think a $10 reimbursement to my account will make up for over $1000 in profit loss?

Still down. No email, no website @ 1630hrs CST.

dreamhoststatus.com – proudly hosted on mediatemple since 2005

need to be compensated financially for this down time… god only knows what emails I have been missing today!!! NOT COOL!!!! also FALSE advertising about 100% uptime… DH should take care of their customers before this gets messy and more serious action is taken against them….

4 hours later, and still no email.

Isn’t there supposed to be some sort of redundancy built in here?

I am so thankful my account with DH is up for renewal in a month! Who here thinks I’m going to renew my account? Anyone? No? Damn f-ing right I’m not renewing it. I’m moving all my sites over to a new host this weekend. I’ve been with DH for low volume sites probably 10 years but I’ve had enough of this shit. Good job DH. May you all join the ranks of the unemployed very very soon. I will have nothing but bad things to say about this company to any and every one who will listen.

Why did Dreamhost indicate that the problem was identified and would be resolved shortly, and then everything went from bad to worse? How about some follow-up communication on what’s really going on?

@Dvorah The email server trying to send them to you will keep trying for 24 hours (or so) and if it continues to fail then the person sending it will be notified. The person sending it will also get notifications that the message has not been delivered but that the server will continue trying for some period of time.

bunu yazan ömer okuyana gömer

What a bunch of self entitled little babies. If these services are so important to you, then you should be hosting them yourself. Shit happens. Deal with it.

OUTAGEHOST STRIKES AGAIN.

Can’t get to any website. Can’t get to panel either to even send a message saying sites, mail is down!

March 23rd, 2012 at 2:34 pmEder Acevedo Says:

@paco says bye bye Dreamhost. No more contracts from me to you.
It is absolutely inacceptable. All my services are down, and still waiting for a solution.
I just pay for another supplier.

I need my sites to run to migration

Esto es inaceptable, otra vezzzzzzzzzzzzz!, es increible que Dreamhost preste este servicio, está claro que llevo muchisimos años con DH y por eso “disculpé” sus pasadas caídas, pero creo que ya es justo …

Acabo de conseguir otro proveedor y necesito el servicio funcionando de neuvo para realizar la migración.

me han hecho meter en demasiados problemas estos meses, ya es justo ….

I hate you dreamhost … you sucks!

Funny how all my websites are down, but while they’re down I seem to be getting emails from my wordpress site asking me to moderate new spam comments.

My sites all went down within the last half hour, three different servers. At least one was fine before that (not sure about the other two as I had no reason to check them). So it doesn’t seem to be getting better.

WHOEVER IS FAVORING AND PROTECTING DH AT THIS TIME

ARE

BITCHES OF DH

Doesn’t anyone else want to get naked?

I’ll start.

PANEL.DREAMHOST.COM IS DOWN!

Way to go. Way to fucking go.

Can you please get your shit together here.

now worse
all down!!

just keeping this from getting buried:

sign here to petition for a formal response: http://www.ipetitions.com/petition/fed-up-with-dreamhost/

OMG!!!!! 4 dedicated servers down… this is tragic. I cannot even access the web panel.

These problems occur too often: honestly it’s not a nice business card.

DreamHost.com isn’t down…….. :)

March 23rd, 2012 at 2:35 pmBarry From Eastenders Says:

@neo Did you get the flowers I sent? I wasn’t sure whether to send them to “Mr Anderson” or not… I hope they got there OK. Cant wait to watch Steel Magnolias with you on Sunday x

Between the blocked email issues and this downtime I can’t even guess at the business lost. Everything for each website I support on Dreamhost is down. The sites, email, Dreamhost Panel, FTP and Mysql. I find that rather remarkable that all my sites happen to be affected by this when it is only a small subset of customers. How did I get so lucky?
I can’t even get in to issue a trouble ticket. Dreamhost should just credit everyone for a whole day of downtime whether the ticket was issued or not. We were considering finding new hosting after the email issue last month, now this. I guess I’ll have to seriously start looking elsewhere for hosting. It’s a shame to because we’ve been with Dreamhost since the beginning. How did things go downhill so fast?

Let’s see if we can get to 1,000 comments!

I’ll jump naked out of a cake!!!

down down down

as soon as I can access my files / I’m switching.

email is mission critical. no excuses

Permanently Flooding with IP spoofing from LiquidWeb :P ?
What about Facebook servers they are still ON?
Dreamhost is a warrior .. He fight for us! :)

Am trying to log a ticket on dreamhost.com, and this site appears to be down too.

this is really not good people…. second time…. please!

I’m not losing thousands of dollars like some people, but I’m job hunting, and my resume is online. When I send people to my resume, and it doesn’t work, that reflects poorly on me. Please get this fixed, and make sure it does not happen again. I’m sick of this, and ready to quit Dreamhost!

First time I’ve had down time with dreamhost and have been with them since 2005. BUT my website is the way 99.9% of my clients book appointments with me. Losing $70 an hour is not pretty guys! PLEASE don’t let this happen again!

Aw, for crumb’s sake. Again? Still? This is really getting old. And the “update” on the situation is not particularly helpful. I speak only for myself, but I surmise many or most of us affected by this outage are more interested in WHEN it will be fixed than we care about WHY it is (still) happening (again). Give us a reasonable estimate of when the system will be back up, won’t you?

I’m sure you are scrambling but this is unacceptable. We’ve been a customer for a long time and the recent issues are starting to cost us a lot in client-trust equity. We can only blame another vendor so many times before our clients start doubting our ability to deliver.

We’ll be moving all critical services this weekend.

@Mandamus Entitled? This is a paid service that claims to have 100% up-time backed by a guarantee. This outage and ALL of the other similar ones over the last few months have been affected everyone from $10 dollar a month shared hosting users to $500 a month VPS users.

This is the fourth or fifth downtime in less than a year. Why am I still here?

Hehe, Love the comments. I guess you all can’t work so are hanging out here.

Hopefully things will come up soon!

Hey Dreamhost. I enjoy having my account on your servers, but I don’t actually host any production sites here, so it’s not too big a deal for me. Just an inconvenience.

@HaLo2FrEeEl defending the un defendable?

its their problem to be a step foward on this issues.
I could just be sewed for this.

Apparently, I’m considered a small subset customer. :(

dreamhoststatus.com will go down pretty soon too lol

Barry From Eastenders Says@

BITCH OF DH

Great, site is down 2 days before one of our big LANs… :\

Seriously, get some ccie R&S and SP guys in there to sort your networks.

How about moving our site to the same server as http://www.dreamhoststatus.com This site always seems to be up…

really bad….

Oops… I mean dreamhoststatus.com isn’t down! :)

My email and sites are down again. The last couple time they went down, I winced at all the outrage pouring out in the threads on these status updates. I’ve been with Dreamhost for over a DECADE and have always seen them a reliable, if a bit idiosyncratic, hosting service. But now… ugh. This is happening so frequently and to so many people (count me in as one of the ‘few’ who ALWAYS seems to get hit) that I’m getting fed up. I’m about to quit my 9-5 job to go indie, and will soon NEED better uptime to run my business. I’m afraid if this keeps up I’ll have no choice to leave a service I’ve loved for a long time, to make sure I don’t threaten my fledgling dream.

While I could live with this for my vanity sites, it’s just is not an acceptable level of service from Dreamhost OR from me. I, personally, don’t want to share the image Dreamhost is creating for itself, and am willing to pay more to prevent that from happening.

I’m not sure what can reassure me at this point. Pull it together Dreamhost, you don’t get another pass. If you go down again before I start my new career, I’m gone. Only uptime, over time, can inspire my confidence in you.

Everything is down — email, website, panel even Dreamhost’s own website. This is a lot more than “a small subset of customers”

I can’t find Google, Google Search is down? What happened? :P

keep the petition alive, repost if you like, etc: http://www.ipetitions.com/petition/fed-up-with-dreamhost/

March 23rd, 2012 at 2:37 pmNathan Holth Says:

My friends website has been down all afternoon, but my website was unaffected. Now, suddenly, my website is down too. Rather than fixing the problem, it seems to be getting worse.

March 23rd, 2012 at 2:37 pmUrine Walker Says:

I like how people waste their time writing comments on here.

I have been a happy Dreamhost customer since 2003, but two major outages within weeks of each other makes me and muy clients… unhappy

Dreamhost seems to be downplaying the severity of this outage. Even their site seems to be down.
Suggestion: new name – “Nightmare Host”

March 23rd, 2012 at 2:38 pmbuythiscomputer Says:

Are Dreamhost engineers just a bunch of handicapped ?

I find it interesting that ***every*** time there’s a problem that affects “a small subset of our network”, **all** of my sites become unavailable?

You’d think that the “small subset” wouldn’t always be the same people.

with a 99.9% uptime guarantee, we should all have free hosting for a while…

but i’d rather pay to have good hosting then have this for free anyway.

:'( finally when I get motivated to do some programming, completely down. My site went down around 2:00PM PST . grrrrrrrrrrrr

oh noes!!! the drama!!111111

they’re working on it, stop whining…

@HaLo2FrEeEl What utter rubbish. Look man, normally when people complain about downtime, I’M the one defending Dreamhost. Pointing out how all suppliers have problems this is to be expected blah blah blah.

But forget about other hosts for a moment, even by Dreamhosts standards, the past few months have been horrendous!. I’ve been here 8 years and its never EVER been this bad, period.

That is why soo many people are pissed off right now, its a scary sharp downward trend.

this makes me want to jump

You are literally killing me. A major business opportunity lost because of this outage. What happened to your reliable service? This year has been TERRIBLE! Slowness, outages, etc. Hate to say it, but I don’t think I can stay with DH.

You mean those pithy newsletters don’t make you want to stay? Any company with this many issues should be issuing credits and kissing my feet. Any suggestions for another host? We should DEF be trending #dreamhostdownagain on Twitter. That might wake their customer service team up.

@forcefield

totally agreed.

“buythiscomputer Says:
Are Dreamhost engineers just a bunch of handicapped ?”

Naah, just a bunch of kids

All of our company web sites still off-line. Massive email communication has gone out directing potential customers to our sites – FAIL!!

March 23rd, 2012 at 2:39 pmBarry From Eastenders Says:

@neo I WaNt TO bE liKE yOu BUt my CapS wONT sTAY on, hOW yoU Do WizARDy?

I HAVE THE SOLUTION TO ALL YOUR PROBLEMS! JUST CLICK HERE:

http://www.dpbolvw.net/click-5620033-10410813

March 23rd, 2012 at 2:39 pmbuythiscomputer Says:

I want to know what dreamhost is doing with our money ? Are employees all millionaire and spend most of the time on vacation ?

Indian and Filipinos seem more reliable than Dreamhost engineers…

I can’t Access my Word Press Blog Is anyone having problems with their blogs?

This is just terrible. Homiemail-sub5 is what we depend on to do business.

Panel down. I see what you did there….

I was up for a very short period of time. Down again. WTF?

Color me one of the “small subset” who cannot reach any of my servers

I moved my main site from DH after the last outage. It was a lot of work, but worth it.

I supposed this is why I can’t update my svn project…

I hate to fan the conspiracy theory flames, however I can’t help but wonder if this has anything to do with the management changes. The major outages seem to coincide with this leadership change.

whole site is down, can’t even logon to dreamhost…. and have a live event im supposed to be covering in 20 minutes…. always the worst timing…

ALL DOWN AGAIN!!!!!!!!!!!!!!!!!!!

I PAY FOR…??????????????

Update Mar 23rd, 2:20pm PST: The uplink which was causing the initial problem came back online; however, when we attempted to integrate it again we experienced network instability which necessitated a reboot of our core routers. WTF

no panel, no web, no ftp, no email

Our undestanding to everyone suffering this situation.
We are offering now free migration for angry customers!
Limited offer! just the time DH is Down!!
Hurry up!
info@inquietudestecnologicas.com

Anyway, Dreamhost was down from 1996 :\
I will love Dreamhost again if they will increase the RAM LIMIT from 90MB to 120MB for Shared Hosting. Thank you Dreamhost :)

Realistically, I will not move hosting companies after this, but this is the last straw. I had better up-time on Godaddy for Pete’s sake.

The management of Dreamhost should be appalled. Someone (or several people) should be fired. I don’t say this to be vindictive, but someone in a leadership position obviously can’t rise to the level of responsibility that’s been handed to them.

It’s sucks for us customers because hosting is associated with a high cost of leaving. It takes a lot of time and energy for us to transfer hosts – which means most of us will stay. But bad press means that Dreamhost will acquire new customers more slowly, and have to pay more for them over time.

This isn’t a one-time incident. When this is over there will be meetings. “How did this happen” “What could we have done different” But the problem needs to be addressed at a higher level. The entire leadership and decision making process is in question, and it needs to be solved if Dreamhost wishes to have a viable future.

If the brass isn’t devoting 110% effort and addressing the larger, overall root causes, they are fools. (Fools I say!)

I have a lot of money invested in my various websites, and I sincerely hope they have some tough decisions to make.

unbelievable. you guys get the prize.

probably the worst day ever for this to happen AGAIN.

dream host, why don’t you try coordinating an event with over a 100 people traveling from around the world to get to you with without email.

I am soo pissed. Get it together or i am gone.

@VRV – lol, that’s why your site is down? douche.

where did u move VRV?

PLEASE SEARCH FOR SOME OTHER HOSTING COMPANY AND LET ME KNOW SO I CAN MAKE A SWITCH.

DH ARE BUNCH OF RETARDS NON PROFESSIONAL PEOPLE AND LOSERS.

HEY DH GUYS AND CEO OF DH

START A PIZZA DELIVERING BUISNESS.

I BET YOU AND YOUR GUYS FAIL TO DELIVER IN TIME.

My site is OK, webmail is down and dreamhost.com is not reachable

@Nick

And? I don’t host anything anywhere that I don’t control that’s that important to my business. If it’s that vital to you, then host it yourself. that way when shit happens, you won’t have anyone to blame but yourself.

Almost 3 hours since it posted. How long will you need to fix all these mess?

hoping to get a few hundred signatures and send this off to someone at Dreamhost:
http://www.ipetitions.com/petition/fed-up-with-dreamhost/

March 23rd, 2012 at 2:42 pmGregory McArthur Says:

WTH I need to update my blog. People need to know that I pissed on the floor!

all my sites down
using outlaws server

Forced Friday off. Awesome.

I think John Galt must have visited the competent people at Dreamhost a few months ago. Time to move to Galtsgulch.com Hosting.

March 23rd, 2012 at 2:42 pmbuythiscomputer Says:

Somebody who could move our websites to Linode or Rackspace would be come millionaire !!!

What is this new CEO doing? He/She needs to first address these ongoing issues with DH. This is one of many issues which have been occurring. Maybe they are having financial difficulties.

10 year customer……this is happening too often now. I like Dreamhost. I really do.

But they must mean something other than the obvious when they say “one of our datacenters”.

i will send my assassin friend ,Ogun Samast, to kill u dreamhost http://www.ogunsamast.com/

dreamhostsucks.com is also down

C’mon, DH. Down 40+ minutes now. The rerouting redundancy for a problem like this should be automatic and almost instantaneous.

Requests get as far as Comcast, then vanish into the Black Hole.

We like Dreamhost, and we get what we pay for. But this is definitely ruining my afternoon.

I would think it would take less than 20 minutes to reboot a router.

Are any sites down for anyone else besides me? :) (kidding).

Hello, we are no email and no website. What is happening? How long to return to
normal?

Regards

This is bad, very bad for my business.

I’ve been a DreamHost customer since Nov. 2006 and there have only been two periods in that time where things have been this bad. With all the downtime and the security breach lately, I’ve been moved to start looking for a new web host, even though it’s such a huge hassle to switch everything over.

I’m glad I don’t use DreamHost for anything mission-critical. Even though I use it only to host some non-critical blogs, my personal website, and some non-essential accounts (mail, backup, Jabber, etc.), I’m still paying money and, when *everything* is down for hours on end, like it is now, I’m essentially paying for nothing.

All my mail accounts, all my websites, my Jabber server, the web panel, and even DreamHost.com is down. What’s going on?!

I’ve always been willing to give DreamHost another chance but this is becoming too much…

Dreamhost, HURRRY UP!!

My business is down because of this. It’s happening way too often since the beginning of the year. Maybe you guys should pay for my loss!

I see a post from Brandon, ostensibly the guy in charge of addressing this issue, at 2:15pm saying “The rest of you can rot in hell.”

I suspect this is someone who is masking as Brandon, as I could put their name and site in this post and appear to be from them. But I would recommend that DH find a way to prevent that misappropriation of their identity; the purported Brandon is only making things worse for them.

Bluntly Dreamhost SUCKS!!!!!!$$!

Second time this week that my site is down andy business is affected with no remorse or credit firm Nightmarehost! Wish you well and I am do bummed you suck!!!!!!!! After 5 years of being with you I wonder what the :))$&$,;; happened to Dreamhost. I hope your business I as affected as mine by how much you Suck!!!!

ohhh… THIS IS BAD!

Dreamhost & LAX Datacenter are moving? What about Facebook, i can’t search nothing with Google right now?

It does seem like there have been a lot of outages in the past few months….

You are RIDICULOUS!!!! I have been 3+ times in the past month or so. Get me back up and i am getting a new hosting site immediately.

We launched a HUGE press release w a major magazine and expected over 15,000 hits on our site to buy tickets to a bridal show I’m producing. Thanks to trusty Dreamhost that is a HUGE flop since they can’t get to my site. Can I bill you for the massive amount of money we are losing? Get it fixed or we may send you bill.

@Cyrus: what exactly are these “high costs of leaving” one ISP for another. I’ve always found the costs (besides my own labor) to be pretty nugatory.

I’m moving over to Media Temple, probably. I’ve got another year or so in my multi-year hosting contract, but it’s costing me more in money and time to deal with these frequent outages than the money’s worth.

I guess, what does one expect for $10 (or less) per month? Hell, I’d pay $30/mo for reliability. And I’d come out far ahead just not having to sweat it like I’ve had to lately.

Hope the tech guys get things sorted soon, then tell us of their migration strategy to minimise this risk in the future.
Please investigate this properly – I realise that stuff happens. It’s what we learn that is important, so that improvements can be made to processes and systems.

PS – can I have my websites and mail back, please?

Dreamhost in the sense you can dream your connection?

I would say check out mah stupid chicken video at jetchicken.com…. but DH shot may chicken down!

all my sites have been down an hour, and so is the panel.

“Sites will be back up” – Dreamhost

:O

It’s a little frustrating to see the problem described as only email in the initial post. All of my customer websites are down right now and my own as well.

this company is a JOKE! How many times must we all deal with aggravated clients and failed launches due to Dreamhost’s inability to maintain a netwrok. I highly recommend we all jump ship from this shit box.

I’m sure all of those posting here are thrilled to be referred to as a “small subset” of your customers, as though that somehow means we shouldn’t be concerned.
It’s upsetting to be thought of as insignificant. We are not insignificant.

March 23rd, 2012 at 2:47 pmBarry From Eastenders Says:

@Troy you’re right nobody should pretend to be brandon… poor guy just lost some dear friends, who would want that.

Holy crap guys.. ok I can understand once, but this is the 2nd time in just a week or two – all my domains are down, no panel.. WTF sort it out!

I moved all my sites that actually mattered away from Dreamhost after the /last/ fuckery, but have to admire the timing that – literally – within ten minutes of starting to work on the poor remaining few, kerplooie. I know there’s a petition, but I’m wondering if we should be pooling together to spring for an old priest and a young priest, a splash of holy water and some fire. Maybe a lot of fire.

Fucking A. Problem has gotten WORSE. Now ALL my sites are down AND my email. It’s been hours. I mean, seriously. Ridiculous. “We’re sorry for any inconvenience.”

Looks like this small subset of clients loves to write comments! almost 1000!

DAMMIT DREAMHOST

A number of our domains are still down. No email, no SQL etc….. and it pisses me off because a whole lot (only DH hosted) recently also became hacked with some uStream redirect hack.

WTF Dream Host? Please please sort this out.

Mail down all afternoon. Now my sites? Why is the problem getting worse, not better?

I cannot access the web panel or my website :(

Seriously? This happens too often with Dreamhost. I am losing ticket sales while you correct the problem that seems to be a consistent one. Time to look for a new host. Sick of it.

I think i am going to try host gator. I’ve heard great things about that host. My sites have been down 3 or 4 times now in the last couple months. Not acceptable.

This is really becoming ANNOYING!

As pissed off as I am right now, the ONE thing I am happy about is that Brandon is updating this status every 20 minutes. I hope he continues to update every 20 minutes as we’d like to know whats going on.

I’ve got a new product launching soon, if this had happened on launch day my business woulda gone down the toilet.

Whatever they’re fixing the problem. I’m loosing money because of it to. Things happen

Stop bitching about your $8 hosting.

Dreamhost rocks, too bad it’s customers are idiots.

Are there any estimates yet as to when things will be back up? This kind of down time is getting expensive.

*
*******************************************************
*******************************************************
*
* MESSAGE NUMBER: 1000
*
*******************************************************
*******************************************************
*

I’m a 10 years customer. Lately, I started to feel stupid to be here. Mostly when compared with Tektonic, where I never have problems, their support takes few minutes to respond…
Yes, after 10 years, I’m leaving.

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

feckit lads, that’s it I’m off: give me me money back and lets pretend we never knew each other.

Gone with the Wind by Dreamhost! :)

P.S. Dreamhost needs:
– high costs for their services
– low features
– slow bandwith
– RAM limits

—————————————
=> BETTER QUALITY for all! ;)

I love this! Just launched a big client site the other day. Some week of marketing we’re having. Hey everyone, check out a site that doesn’t work! Can’t wait to have all of those visitors Not visit our site again!

Wow, this is so incredibly bad. If one of the servers I maintain goes down for even a few minutes heads roll, the only Dreamhost thing which seems to be even working is this place and it’s not even hosted by them.

As for ‘only a small subset’ – your panel is down, it’s affecting everybody.

This just in from DreamHost:

“Call your favorite Pizza joint, order a pizza, grab some beer (or whatever) and kick back – it’s going to be a long night…. ”

:)

Look at the bright side – Pizza & Beer.

gimme one thousan $ cus this is comment #1000

I think we broke 1000 comments.

Since all our sites are down, we may as well be here.

i m in Turkey and can not sleep…

It’s down for me currently as well. Everything. SSH, web, mysql etc.

Well, I’ll chime my 2cents in here too (maybe I’ll be comment 1000 and win a prize; like a working website)

everything has been degraded since this morning, and 100% all stop at around 3pm eastern today…

sucks…

Yes, I notice their last update was over 2 hours ago.

March 23rd, 2012 at 2:51 pmNathan Holth Says:

Now my email is down too!

Good job, Adrian. I suggest everybody signs it.

congratulations Rogerup you get the one thousand

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

3rd outage this year. Zoho CRM was down an entire day a couple of months back so there certainly are risks of being entirely web based. Dream Host seems pretty nonchalant about getting it back up so we’re definitely looking into other hosting.

I’m not an IT guy at my workplace. But I do perform a lot of IT shit here though because I have the knowledge and background. You can call me the official unofficial IT guy here. And, you better thank ()!$U%$sake that I’m not free until later this summer, because my number main priority, the moment I have the time, is to rid you !$)(U%’s from hosting our e-mail and website.

@Mandamus Host it yourself!? Hahaha…sure you are! Hosting stuff “yourself” is myth. You will always depend on someone for your needs. You depend on the manufacturer of your server to make reliable equipment, you rely on your communication company to provide a reliable connection and the people who make the software. Those are no different than this. If the level of dependability drops then you complain and ask them to change, and if they don’t then you find someone who can provide what you need.

March 23rd, 2012 at 2:52 pmBarry From Eastenders Says:

The rain in Spain falls mainly on the plain

This has happened once too many times now. What is a good alternative to these clowns?

Site down and I can’t even login to Panel to report it. Along with all the bouncing mail and other nonsense this is really making life hard! Haven’t gone a week without some kind of issue for months now.

Gahhh… All me sites be down!!!!

………………………………..__………………………………………….
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

Merdan uyumak için erken daha sabaha 8 saat var

I have decided to start a new business…. moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

they covered.that

Looks like the big problem is internal to Dreamhost. tracert to my website gave me this:
11 2098 ms * * ip-66-33-201-114.dreamhost.com [66.33.201.114]
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.

I never thought I’d see the day when Twitter was more stable than Dreamhost *sigh*

@Matt

Not all of us are paying only $8, some of us significantly more- to a company advertising 99.9% uptime. With advertising and marketing like that, coupled with substandard support, the users here are understandably perturbed.

What do you recommend?
http://mediatemple.net/
http://www.hostgator.com/
http://www.bluehost.com/
?

I can’t assume the risk of being into this “small subset” problem.

HURRAY…. Servers are UP!!!!!

Just kidding….. :)

I think its really funny that dreamhost.com is down. lol

I have 6 different web sites down. FIX WHEN?

…………………………………………………………………………………………………………………………………………………………………………….
Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.
…………………………………………………………………………………………………………………………………………………………………………….

March 23rd, 2012 at 2:54 pmHeetan Patel Says:

I’m severely disappointed in DreamHost. I use my site to live stream surgeries to my patients’ families (SSL of course). Just finished wrapping up a leg amputation (post doc) and sadly the patient’s family was unable to view the procedure. I will be taking my business elsewhere.

Connectivity issues with one of our datacenters

IS LIE

THIER ALL DS ARE DOWN.

I just hope there is no data or email loss with this…!

I usually recommend dreamhost to friends, but really, not anymore. Problems are cropping up over and over, and what was once an excellent service, is now getting worse all the time.

Please Dreamhost, fix this! I love being hosted with you guys, but this is seriously getting on everyone nerves… :(

SOPA, PIPA & ACTA affected us daily and Dreamhost clients!?

I would like a prorated refund for every minute of downtime in 2012 so far. What am I paying for?!

Seems to be cascading.

All my clients on shared server account are down. VPS that was up earlier is now down.

DH forum down.

March 23rd, 2012 at 2:55 pmBarry From Eastenders Says:

whats @neo up to?

one only datacenter ?

Email is down and website is down. Karma will get you!!!!!!!!

Hi, I have 3 sites down actually.
anyone knows if there is the real possibility of loose all my files on Dreamhost servers?

Move all to server which this dreamhoststatus.com located. It never down (hopefully)

The correct meme: is GODDOMIT FRONK!

I’m gone, I hope I can get my money back, this is ridiculous, 1 outage like this a year is bad, 2 is bad luck, 3 in 3 months is incompetent!!!!

this has hurt my business

I wonder: Will DH compensate me for all the lost sales today on my e-commerce site?

This is beyond frustrating.

I’ve been fighting a serious hack on fire factor.net for several days now. It is only getting worse, and I have concluded the attack is coming from the server end. Please get this fixed. I’ve been a happy customer for a long time, but am losing my patience too :[

March 23rd, 2012 at 2:56 pmJohn from Spain Says:

Again? Buff. No more. I leave right now. Anyone can tell me where I find better hosting?

I’m cool and understand this kind of things happen. Have been with 10 other hosts and guess what: It happens everywhere!. Got many clients running on DH and after 11 years I won´t change you guys. Appreciate the effort. For those insulting: Grow up, and go somewhere else without winning.

March 23rd, 2012 at 2:56 pmBarry From Eastenders Says:

@Tom Huff in a huff

I can’t play World of Warcraft anymore :(
Please … i need more bandwith.. Ty

Thank Goodness…. Dinner time right when servers are back up!
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
..
.

..
.
.
.
.
..
.
.
.
.
.
.
.
.Really… they are….
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
..
.
.
.
.
.
..
.
.
.
.
.
.
.
..

.
Just kidding…. :p

Their new CEO needs to step up and fix this or step down so a real/qualified person can finally fix DHost.

Yes, I noticed the dreamhoststatus site is always up? Where is it hosted?

virtually every site i have on here and my clients sites are either not responding or slow or 404ing! guys this was last straw but i cant even load your website to transfer my domains away! GRRRRRR!!!

all my sites are down! :(

Can we get some hosting credit or something? at least?!

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\………

March 23rd, 2012 at 2:58 pmNo more Saul Says:

DH thanks for all the efforts to put forth to resolve. Long term I’m betting we will have a much more resilient hosting service as a result of this outage. Best of luck.

March 23rd, 2012 at 2:58 pmPGP_Protector Says:

@arteweb you won’t loose anything if you backed it all up :)
I had just done a full backup last week due to the php hack that hit some.

March 23rd, 2012 at 2:58 pmgreenfishfiles Says:

Everything is down – email, website, webmail, DH panel! My boss won’t let me join in the pre-weekend drinking unless everything is back up and running. Damn you DH! WTF is going on???!!!11!!!

Son of a bitch

I have decided to start a new business…. moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

Wouldn’t you know it? My site’s down, too.

I’m leaving Dreamhost. Right now, I’m looking at a hosting site called “WebFaction.com”. It came highly recommended for webcomic artists trying to get away from Dreamhost.

We made it to 1,000 comments!

I basically have free hosting because of how many referrals I’ve sent and I’m preparing to leave now. I need my sites and svn to stay up. This is just getting annoying. Although I have to admit, taking some time to read through these comments made me feel better, funny stuff. :)

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\……..
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

kinda stinks that it’s been such a long outage….

My whole site is down. We have a portal that has all our quotes that we need to deliver and we can’t access them.

This is costing me too much money…

Come on, this really is bad. Several hours of downtime?????????????????????????

@John from Spain Says: “Anyone can tell me where I find better hosting?”

Try Dreamhost — http://www.dreamhost.com/

:P

That is a great observation.. you are correct their status site is always up.

March 23rd, 2012 at 2:59 pmEverything is Brilliant Says:

lunarpages.com hosts some of my other sites. They’ve been very reliable for the last 6 years I’ve been using them.

Also, this blows! This downage also effects my apps that are pulling data from the site. We’re switching to Heroku very soon.

I have 3 websites down right now with clients yelling they heads off over the phone…

and is http://www.dreamhost.com working for anyone????

Seems like their own website is down as well….

Looks like its time to move to somewhere more reliable.

Is there a way I can embed a AdSense add into this comment? I could use the hits. ha.

But really man. Get this shit going.

I wonder what is happening with Dreamhost.

Many problems in a short time.

It was not so, but a few months became worse hosting the market.

I want a discount on my account

“Again? Buff. No more. I leave right now. Anyone can tell me where I find better hosting?”

http://www.names.co.uk
http://www.squirrelhosting.co.uk

http://www.dreamhos…. no wait not that one!

Sites are down, webmail down

.
.
.
.
.
Can anybody suggest DH a good Cable Guy / Network Engineer ?
Do they have financial difficulties, lets guy support them by giving your coin.
.
.
.
.
.

not yet corrected

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\………
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

@Matt Dreamhost rocks?, you on crack dude?. It’s been CrapHost the past few months, what planet are you on?. Btw not all of us are on the cheapest $8/mo option either and its affecting those of us paying more as well, so enough with the “oh its cheap so you should expect crap service” (even though it was way better before) nonsense.

Anyone here tried Host Gator?

Anybody want to play Naked Twister?

this type of outages are driving me crazy dealing with customers…he wants to move his domains to another hosting, so thank you for making my life difficult, and making me lose money.

@Cicada I’d rather have my sites up all the time than hosting credit. But if you open a ticket and ask for one they will oblige.

I have 25 customers pissed at me!

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,.
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,……………….
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

We just launched our fundraising campaign today, and people can’t find our site. :( This is frustrating.. please be back up soon!

Mother earth, just swallow me.

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………,
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

I wonder why only customers have to pay for this break downs? DH loosing reputation sure but they have to pay fine to customers affected. Don’t you think? It would be more motivating for them

ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

We’re sorry, but the rest of your posts have been moved to dreamhost.com!
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………. END

*
*
* “congratulations Rogerup you get the one thousand”
*
*
* No, my message was 999!!!
*
*

O M G I C A N T H A N D L E T H I S ! ! ! ! I L O S E $ 1 B I L L I O N
E V E R Y M I N U T E D H I S D O W N T H I S I S T O T A L L Y
U N A C C E P T A B L E ! ! ! ! I A M G O I N G T O R U N M Y S I T E
F R O M Y O U R G / F B E C A U S E I H E A R D S H E N E V E R G O E S D O W N ! ! ! ! M U A H A H A H A H A ! ! ! !

WTF this is not cool can someone plz FIX!

Do your hear about CDN System (Content delivery network)?

http://en.wikipedia.org/wiki/Content_delivery_network

I have now started a new business…. moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

Carajo , me cago 1000 veces en la leche agria de la concha arrugada de su reputa madre que los parió !

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

OMG!

Barack Obama here… the White House website is hosted on Dreamhost!!! What now????
This can’t be, the terrorists have WON!!

40 min isn’t updates

Last update is 40 minutes ago, whats happining?

@Jatemack thanks; as would i! — ill be sure to — luckily my sites aren’t too critical to be up — but for everyone that does business through their sites, it’s increasingly frustrating.

I do run a small forum/community off of my site and it’s really frustrating to get bombarded with questions from all the members on what’s happening.

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

Because there IS no way…

You know? I wonder why I work sooo hard at prepping changes to my customers’ sites to happen in the final 10 seconds instead of bringing their sites down for 3 hours to make a change? Ya think they’d notcie or care today?

Credit me. Credit me WELL. I pay on time and I pay in full.

I EXPECT redundancy. Sorry, I EXPECTED redundancy.

Redundancy now means continue to be patient and expect a different result.

Like losing my customers is the different result I get this week?

Out of hand DH. Unacceptable.

March 23rd, 2012 at 3:02 pmOh Snap It's Down Says:

Wow even my DNS are down… this is entirely too lame.

Ok, so my sites have been down for well over an hour with members questioning me “What the ***k is going on with the site?!?!”.

Prove to me I should stay with you guys, my hosting has gone down way too many times to even care about staying with you guys.

down down down

………………………………..__…………………………………………
………………………..,-~*`¯lllllll`*~,……………………………………
…………………..,-~*`lllllllllllllllllllllllllll¯`*-,………………………………
………………,-~*llllllllllllllllllllllllllllllllllllllllllll*-,…………………………….
……………,-*llllllllllllllllllllllllllllllllllllllllllllllllllllll.\…………………….. …….
………….;*`lllllllllllllllllllllllllll,-~*~-,llllllllllllllllllll\…………………………..
…………..\lllllllllllllllllllllllllll/………\;;;;llllllllllll,-`~-,……………………. ..
……………\lllllllllllllllllllll,-*………..`~-~-,…(.(¯`*,`,……………………..
…………….\llllllllllll,-~*…………………)_-\..*`*;..)……………………..
……………..\,-*`¯,*`)…………,-~*`~……………./…………………
………………|/…/…/~,……-~*,-~*`;……………./.\………………
……………../…/…/…/..,-,..*~,.`*~*…………….*…\……………..
…………….|…/…/…/.*`…\………………………)….)¯`~,………………
…………….|./…/…./…….)……,.)`*~-,…………/….|..)…`~-,………….
……………/./…/…,*`-,…..`-,…*`….,—……\…./…../..|………¯“`*~-,,,,
……………(……….)`*~-,….`*`.,-~*.,-*……|…/…./…/…………\……..e
…………….*-,…….`*-,…`~,..“.,,,-*……….|.,*…,*…|…………..\……..
……………….*,………`-,…)-,…………..,-*`…,-*….(`-,…………\…….
………………….f`-,………`-,/…*-,___,,-~*….,-*……|…`-,……….\……..

goddamn

@Random Emosexual

GO FUCK YOURSELF

EVERY TIME DH GOES DOWN THIS FUCKING Random Emosexual MORON COME FUCKING AROUND

GO FUCK YOURSELF

oh and Andrew, we’re all impressed

Shoot, this is quite a predicament. It wouldn’t be nearly as bad–though still an inconvenience–if I at least had email access… all of my email is broken right now!

WTF is wrong with you people? For real…

if it’s affecting such a small amount – why is dreamhost.com itself down?

seems like the server starts working again whenever I complain. So please acknowledge this official complaint Dreamhost. You suck, blah blah blah, I am losing money, blah blah blah, My email and store are down, blah blah blah. Fix it NOW!

Business is booming…. moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

wow all my sites and their main website is down….

Third major downtime in past 4 months. Seriously what is up I know some people have to be losing money like me.

this is so weak

Look, I can tell the future….
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
……………………………………………………………………………. Poof…. your website is still gone.

time to move to rackspace

I think Dreamhost needs to come up with some incentive for these clients to stay.

Of course… I guess, on the bright side, the more people that leave today… the more that frees up the server bandwidth for the clients that stay.

Just sayin’. ;)

Where is my ice cream?

What’s the latest? Can you please send a quick update?

maybe their status site is hosted at rackspace ;p

I should have read his profile before: http://www.linkedin.com/in/simonanderson

‘Consulting to and investing in early-stage disruptive internet and digital media ventures.’

I’d say he’s successful on the ‘disruptive internet’ part.

For God’s sake,

My server stopped all functions. I need this site up Urgent!

What’s going on guys? All my sites are down, email down, Dreamhost panel is down. No contact with Dreamhost and absolutely no useful information here. Very very poor.

the worst part is a bunch of wordpress sites are probably infected with malware again and you guys dont even realize it yet because you cant even connect to your malwared up site. mine was. this all started last data outage. today within a tiny window of me not being on the computer i got malware again, then the data center went down again. coincidence i think not?

I was clean on Sucuri yesterday. I run BPS Security, WP Secure, & WP File Monitor.

So ridiculous.

After all this, I am surprised how wordpress still officially recommends Dreamhost! What is going on???
http://wordpress.org/hosting/

Can we get the number of comments to… OVER 9000?!

Quick! Hurry up and wait!

readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..
readya ready readya readyaaaaaaaaaaaaaaaaaaaaa. ..

Small subset what a load of bs

Sorry Guys, no notification updates because i felt a sleep.
Be patient……

It would be nice when we pay you, to give us a bonus for all those Connectivity issues that you have all the time!

I have two websites down right now so I’m a little ticked off but I have to laugh at all the facepalms from Andrew

srsly? more downtime? you guys need to invest in your infrastructure. this is a joke. it has not been fixed and it has been 3hrs… c’mon guys, if you’re going to have downtime, at least make it the weekends so i don’t have to get yelled at by my customers.

Maybe a Bank-to-Credit Union style exodus will grab someone’s attention.

1141 responses. Web Panel is down.

Sounds like more than just a ‘small subset’.

___ _ _ _ ____
| | |\/| |___
… | | | | |___
,’ ‘. ___ ____
; ; | | |
| | | |__|
| | _ _ _ ____ ____ _ _
| _, ,_ | | | | | | |__/ |_/
‘´ | | `’ |_|_| |__| | \ | \_
| | _ _
| | .|_|_,’ | ,–.
| | | | : |
| | `|`’`-._| : |
| | | | : |
| | | | : |
| | | | .——-. : |
| | | | ,| _O | : |
| | | | ||_,’ | : |
| | | | `| | ;…:fsr
| | | | | | /.–. \
| | |` | | | |’–‘ |
‘-‘ `–‘ ‘——-‘ ‘—–‘

Video from the Dreamhost data center: http://youtu.be/dsx2vdn7gpY

March 23rd, 2012 at 3:08 pmBarry From Eastenders Says:

Who is this Superman?

March 23rd, 2012 at 3:08 pmEverything is Brilliant Says:

@Paul Your portal of quotes is down? NO0O0O0O0O0O0O!

I have two distinct accounts at DH, and some friends also do. Some days ago WE ALL received a message that we should change our passwords due a potencial use of OUR passwords. The funny is that everyone received the same.,..Conclusion, they did not know what happened and used this MESSAGE to everybody……..

OMG!!! My new business moving users from DH to anywhere else rocks! Thank you free enterprise!

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

This is the funniest site on the internet right now…

Will anyone dress up as a buttercup for me?

.,c=””” _JLccccc,
,r=” ,,-‘ “=c
,J” ,c$?$ $ `$c `=,
,c”‘ ,J”” J ( j’ “$. “c,
?????????????””””””???”=hcJ$” JL `r `$r “$q
`.`.`.`.`.`.`.`.`.`.`.`.`J” j’L `L \ ?c “g
`.`.`.`.`.`.`.`.`.`.`.`.J j’`?c ?. t `$. `=hc,;c???
`.`.`.`.`.`.`.`.`.`.`_,$ z’.`.?L $ `h “=c, ,cJ????””`.`.`.`
`.`.`.`.`.`,cPP?”””‘? J ;,?.`z$$h`”$.’L3, `L $ “” $`.`.`.`.`.`.`.`.`
`.`.`.,c??” ,f $ / $.`.,,,`”=`?h`?$c `r 3r .$`.`.`.`.`.`.`.`.`
“hcr=” j F j P.`$$$h3$,`.`???$c `cc$r.P.`.`.`.`.`.`.,r`.`
`.$ ( J d$ j’.`?.$$$$`.`.,`.`=”$c.,cP”” $.`.$??”??$hc$”.`.`
`.`h ( $ $$ $`.`.`.””””`.J:`hJ?$h,`.$ JF.`$” .$.`.`.`
`.`.?. `j’ ($$ $`.`.`.`.`.`f.`?$$$$$$r’3 J”`.J” \ .P`.`.`.`
`.`.`?. $ ($’ $`.`.`.`.;`.`.`.`”?$??”`$ .F’,$F L .P.`.`.`.`
`.`.`.?. j’ ($’ $`.`.`.`”=cj’.`.`.`.`.`,P $’$l?F $ j’`.`.`.`.`
`.`.`.`?, `$. ? ‘$`.`.`.`.`.`.`.`.`.`.`,$ $$j’j’ $z?.`.`.`.`.`
`.`.`.`.?, ( $ j’`.`.J$I?$c,.`.`.`.`.J$F $cJ 3F`.`.`.`.`.`
`.`.`.`.“h j’ $h`.`.?6$cI$9;`.`.`.c$$” .$” L .P”.`.`.`.`.`.`
`.`.`.`.`.`?. $ J$?,.`.`.??i???`.`.`.`$F.P$$ ? J?.`.`.`.`.`.`,P
`.`.`.`.`.`.”h. $ J$$.?.`.`.`.`.`.`.`.`,$$,c’`$ L,P.`.`.`.`.`.`.,F
h.`.`.`.`.`.`.?h J J3$$.`3`.`.`.`.`.`.`.J$$”$ $ J”`.`.`.`.`.`.`,P
`h`.`.`.`.`.`.`.?h .F J'<$F.`.?h_.`.`.`.,c$$$$$ $ ,P J?`.`.`.`.`.`.`.,P
$.`.`.`.`.`.`.`.?$ J' <$`.`.`.`?=hc=?"'J$$$$" $ $.P`.`.`.`.`.`.`.`,P
$`.`.`.`.`.`.`.`f $ <$`.`.`.`.`.`.`.;$$$$ F $?.`.`.`.`.`.`.`.,F
?.`.`.`.`.`.`.(f $. j'`.`.`.`.`.`.`.$$$P JF JF`.`.`.`.`.`.`.`,F
",.`.`.`.`.`.`$'F J)`.`.`.`.`.`.`J$P Jl$$?.`.`.`.`.`.`.`.,F;
"i`.`.`.`.`.`? $ ($.`.`.`.`.`.`.z$" J$"J"`.`.`.`.`.`.`.`,P } j
?`.`.`.`.`.`.?` P.`h`.,c`.`.`J" ,' z$"'J".`.`.`.`.`.`.`.,P j f
?.`.`.`.`.`.`h F.`$'z?.`.`,",$" zP"`.`?`.`.`.`.`.`.`.`,P , j'
?`.`.`.`.`.`.??'.<$$.`.`.J$"j'c$.`.`.`.`.`.`.`.`.`.`.j' $ j .
$.`.`.`.`.`.`.`.$?`.`.`.`.,$?.`.`.`.`.`.`.`.`.`.`.`JF .F J"L
$`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.J$ $ .P k
`h,c.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.,$$$ `h P
,P"`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`,$"$$ $ $
,P.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.;$F $$r ?c, _c"
,P"`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`;$'F $$ ""
.$"`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.,P $ `$$
J?`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`,$ $ "=,_
,P`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`$$ ? "=cccc"
,$"h`.`.`.`.`.`.`.`.`.`.,cc,`.`.`.`.`.`.`.`.`.`.`.`.JP$. `t -c 3,
$`.3`.`.`.`.`.`.`.;.`.`$,c,3`.`.`.`.`.`.;;`.`.`.`.`;$ `L $ 3. ?.
$$iJ`.`.`.`.`.`.`;;.`.`$?;t$).`.`.`.`.`;;;`.`.`.`.,$' $ `L $ $
?$F;`.`.`.`.`.`.`;;.`.“$iP"`.`.`.`.`.`;;:`.`.`.`.$$ $ 3. $ 3
`h;;,.`.`.`.`.`.`?;;`.`.`.`.`.`.`.`.`.;;3.`.`.`.`,$$ j'L $ J" $
?;;;;,.`.`.`;;.`$;;;.`.`.`.`.`.`.`.`;;;P.`.`.`.`$$F j ?. `"??" v'
`h;;;;;;,;;;;).`?h;;,`.`.`.`.`.`.`;;;;$`.`.`.`.f'$F J ?, .$"
?c;;;;;;;;F`.`.$h;;;,`.`.`.`.,;;;;;$'`.`.`.`f $) $ `"?????c=" 4r
?c;;;;;;F.`.`.`?y;;;;;;;,;;;;;;;iP`.`.`.`.J' $ $ j'
`"?jj$F;.`.`.`.`$y;;;;;;;;;;;jP".`.`.`.`;$ $ F j'
3;;.`.`.`.`.`?hijjjjid$?".`.`.`.`.,f $ F j'
h;.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`,;$' ;F L P
?h.`.`.`.`.`.`.`.`.`.`.`.`.`.`.,;;$ $ ? ,cc $
`$;`.`.`.`.`.`.`.`.`.`.`.`.`.`,;;;$ $ " ?. z"
?;`.`.`.`.`.`.`.`.`.`.`.`.`.;;;;;$ `r h.,zJ"
h;.`.`.`.`.`.`.`.`.`.`.`.`;;;;;;$ ? `$
`h;;;,.`.`.`.`.`.`.`.`.`.`;;;;;;$ $ .F
$;;;;;;.`.`.`.`.`.`.`.`.,;;;;;;$ "hcc$ $
`h;;;;;,`.`.`.`.`.`.`.`.;;;;;;;$. j'
?;;;;;;;.`.`.`.`.`.`.`.;;;;;;;9L, J'
?;;;;;;;`.`.`.`.`.`.`.;;;;;;;;h "cc,__,c"
$;';;;;,.`.`.`.`.`.`.;;;;;;';9r
$;`.`.`.`.`.`.`.`.`.;;`.`;`.`?c.
$;.`.`.`.`.`.`.`.`.;.`.`.`.`.`."c._
h;`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`."?c,
?;;.?;`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`."=c_
$;.`$h.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`."c
?;;`.$)`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`."h.
h;`.`?`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`?.
`C;;`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.`.?c
$;;,.`.`.`.`.`.`.`;;.`.`.`.`.`.`.`.`.`.`.`.`.`.h
?;;;;`.`.`.`.`.`.;;;;`.`.`.`.`.`.`.`.`.`.`.`.`.`h
h.;;;;`.`.`.`.`.;;;;';`.`.`.`.`.`.`.`.`.`.`.`.`?L
?.`;;;;.`.`.`.`.`;;;;;`.`.`.`.`.`.`.`.`.`.`.`.`.3.
?`.;;;;`.`.`.`.`.;;;;`.`.`.`.`.`.`.`.`.`.`.`.`.`$
$`.;;;;,`.`.`.`.`.;$`.`.`.`.`.`.`.`.`.`.`.`.`.“L
$`.`;;;;.`.`.`.`.;$`.`.`.`.`.`.`.`.`.`.`.`.`.`.$
$;`.;;;;;.`.`.`.;9`.`.`.`.`.`.`.`.`.`.`.`.`.`.3
$;,`.`;;.`.`.`.;?`.`.`.`.`.`.`.`.`.`.`.`.`.`.3
$h;;`.`.`.`.`.;;L.`.`.`.`.`.`.`.`.`.`.`.`.`.3
$'"$;;;;;;;;;;;j$.`.`.`.`.`.`.`.`.`.`.`.`.`.$
$.“`?;;;;;;jjJ?$.`.`.`.`.`.`.`.`.`.`.`.`.`.$
J'`.`.`"hiii?";;;?h`.`.`.`.`.`.`.`.`.`.`.`.`.$
,$`.`.`.`.“?i;;;;;;$`.`.`.`.`.`.`.`.`.`.`.`.`j'
c"`.`.`.`.`.`.?$i;;;;$`.`.`.`.`.`.`.`.`.`.`.`.`$
.P`.`.`.`.`.`.`.`.`?iii9`.`.`.`.`.`.`.`.`.`.`.`.`f
z?.`.`.`.`.`.`.`.`.`.`.`3).`.`.`.`.`.`.`.`.`.`.`.j'
J'`.`.`.`.`.`.`.`.`.`.`.`?).`.`.`.`.`.`.`.`.`.`.`.3
j"`.`.`.`.`.`.`.`.`.`.`.`.`.h.`.`.`.`.`.`.`.`.`.`.`.3
,P`.`.`.`.`.`.`.`.`.`.`.`.`.`.$.`.`.`.`.`.`.`.`.`.`.`.

Getting kind of sick and tired of the quality of service lately… If it’s not network issues, it’s nefarious sites using up all the resources on the shared server. I’ve been a huge DreamHost fan for going on 7-8 years now, but I’m starting to feel like a battered spouse that keeps making excuses. It’s time for real change.

Please make quality of service your main focus. Slow service(or in cases like today, no service) is all too common with DreamHost in the last couple years.

But, given the effort you have been showing lately, I doubt anyone from DreamHost will bother to read all these comments by upset and concerned customers.

Can we get this fixed SOON please?

You aren’t hosting personal blogs here, but people who run businesses and need their sites up and running.

Where are the updates? Its been almost an hour since the last one.

I not berry happy right now :(. I had a major account question our professionalism because of this. I am going to check out slicehosting in other places while we wait. This is just happening too often these days.

Having mail down in the middle of business day for hours is a major problem.

What makes it worse is Dreamhost not notifying me or others affected about it.

And I find it a little disingenuous to read from Dreamhost that it’s “ONLY” affecting a small number of customers. That sure makes me feel valued as a customer!

I’d like a credit for this disruption of service.

Probably time for a new web host provider.

Our website has been down for the past two hours and our email is shut down as well. When do you think things will be back up and running? We have customers that are trying to place orders…

SEHAWK server and all of its “small subset customers” stalling down!!!

gokhan bana bi server soyle de bu sorunlari yasamayayim..

So… uhm… yeah. I was developing a new website on here, and I’m starting to seriously reconsider my choice… I was with 1and1 for years, and they suck Bawlz, but my site never went down for hours on end… Just sayin’…

March 23rd, 2012 at 3:10 pmRafael Goulart Says:

This definitely is NOT the year of Dreamhost. I’m seriously considering move to Linode, which hosts this blog.

SEAHAWK server and all of its “small subset customers” stalling down!!!

It’s after 5:00 Somewhere…. Everyone just W A L K A W A Y

{((((((}
{)))))))))
((((( _ _))
)))))) . (.((
(((((( >)))
_)))))) – /(
.-‘ (((((.–‘ )))
.-””’-._ _.-‘ :: (((((
.` ‘-. .-‘ _.-” :::)))))))
: ‘:-: _.-‘ :’ .:::((((((
: .: .:: : : ::::)))))))
:. ..:: .:: : ‘ ::::((((((
:. (::: .:::: : : o ::)))))))
_.-. :. \/ :-‘-‘-‘-‘`’-.-‘`:: :((((((
__.-‘ _\ _ :. \ / ))):. :))))))
( .-‘ ;;’-.-”’ ”:. : (((((:. :((((
“`’-..-.;._ _.-”-.\ : ))))):. ))))
__.-‘ _\_.`’ \ : (((((((:. ‘(((
( _.-‘ ;; : :. ‘—.__
`”’-….;;,,,,,,,,,,,,,,,;,___: \:..b====’

So basically, an uplink goes down affecing “a small subset of customers”. It takes several hours to reroute around this and service is never restored. Then the uplink comes back up and reintegrating it takes down all of dreamhost.

@Nick

The last time something like this happened to me was about 2 years ago. I had a co-location in Vancouver and a main power switching station that fed that datacenter blew up and caused a massive fire. Shit happens. No matter where you go, you’ll just trade one set of problems for another.

The second line of this post seems really wrong at this point… you have to get into it in detail to discover it’s a major failure now…

I’m done as soon as I can organize the transition. Has anyone heard anything about Hostek or Rackspace?

Anybody offering free hosting for a month on a switch? My bank is closed Monday, but I don’t want my site to be =/

please update us on the status

still fucked.

they seem to have outsourced support of non-english-speaking morons who don’t know a thing about their products. now their IT department in sin the same predicament.

Jimmy, that’s cool. But it ain’t as good as Teletext Porn. http://drx.a-blast.org/~drx/projects/teletext/index.en.html

SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
SEAHAWK server and all of its “small subset customers” stalling down!!!
and spamming does not relax me @ all

You know the servers have been down too long when people start loosing the plot and making ASCII pictures.

bummer.

March 23rd, 2012 at 3:12 pmNathan Holth Says:

Why has a time estimate for recovery not been provided here? Has Dreamhost never done a “reboot of our core routers” before?

O come on guys it’s Friday, this is just a legit excuse for us to bounce out early, beside bitching ain’t gonna fix it any faster… I still give DH a A+ for the money the service can’t be beat..

March 23rd, 2012 at 3:12 pmPatrick Rosenheim Says:

all down

again

wow

just, wow

WHATTT? ALL MY SITES ARE DOWN?????

fucking a

Oh, man. Makes for a helluva slow Friday when you don’t get email all day. Down now for 3.5 hours. Really hard to run a company when this keeps happening…

goddamn it ascii just doesn’t work here.

Of course it doesn’t – this is DH. Nothing fcuking works lol.

March 23rd, 2012 at 3:13 pmJohn from Spain Says:

small subset? Come on! Tell us the truth.

down on albireo

Nearly an hour since they even bothered with an update, and for the last ten minutes this emergency blog has been inaccessible, too.

What a bunch of fucking clowns.

All Dreamhost sites down. All Dreamhost email down. Yikes!
1st attempt to post here returns Bad Gateway error.

My entire site is down, not just mail, MySQL and “internal” services.

Time to change my hosting provider…

“A formal response from Dreamhost about their recent run of bad service”
http://www.ipetitions.com/petition/fed-up-with-dreamhost/?utm_medium=email&utm_source=system&utm_campaign=Send%2Bto%2BFriend

I really think this is an important cause, and I’d like to encourage you to add your signature, too. It’s free and takes just a few seconds of your time.

please share & repost this…maybe someone will take notice…

Wake up Jeff!!!

@charmaine rackspace is very good. I’m considering them or Amazon.

ı think engineers watching an nba match or something else?? playing cards…

Seriously, DH has an awesome panel.

OK, enough, I am leaving, moving to Hostgator they even give me a 25% discount for moving from DreamHost with the code COSMOPOLITAN

I really liked u DH, but 2012 seems not to be your year and I have to keep my business running.

How to jump start the economy…. start a business moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

Amazing how the status messages always suggest it’s such a “small” subset of users, yet it just “happens” to be the datacenter I am using over and over again! Where is this datacenter, some alley in LA? Luckily, we are moving to our own datacenter, so GOODBYE and GOOD RIDDANCE. I will definitely not be recommending Dreamhost to anyone, and will in fact make sure they DO NOT use you.

Perhaps a little less time crafting the witty emails will help focus on the servers?

I have been using Alentus for over 10 years now for most of my sites and they have been great except for a very few minor issues. http://www.alentus.com I have VPS at their sister company too, websitesource.com. I suggest giving them a try if you are really looking to move. I did and have never regretted it.

This is pure BS. It doesn’t take an hour to reboot a core server. Even when our entire university is fritzed, it doesn’t take more than 20 minutes to bring every last server for all 8 colleges, the library, the research foundation, and the dorms back up. Fess up to it, DreamHose (sic). You hit the wrong button and how you’re trying to unscramble the eggs.

Down for me too. Holy cow, 1200 replies. They better get on the ball with this, unless we’re talking about a backhoe accident outside their data center.

Dreamhost….seriously?!

This is seriously impacting my business. I had some patience earlier in the day but now it is gone. Also, mentioning that a only a small subset is affected only enrages those of us in the small subset. I don’t care that it’s small- it’s impacting me- and that’s not something to belittle.

high quality service

What do you recommend?
mediatemple.net
hostgator.com
bluehost.com
rackspace.com
?
I can’t assume the risk of being into this “small subset” problem.

Feverishly

WTF???? is this a weekly thing now?

This is seriously impacting my business. I had some patience earlier in the day but now it is gone. Also, mentioning that a only a small subset is affected only enrages those of us in the small subset. I don’t care that it’s small- it’s impacting me- and that’s not something to belittle.
my site and mail are down. Big deal.

Wow! http://www.dreamhost.com is even down. Haven’t seen that one happen before.

I just became a millionaire with my new home business moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

Mission critical

From Wikipedia, the free encyclopedia

For the computer adventure/strategy game, see Mission Critical (video game).

Mission critical refers to any factor of a system (equipment, process, procedure, software, etc.) whose failure will result in the failure of business operations. That is, it is critical to the organization’s ‘mission’.

Our site is down too.

here is something to watch while you are waiting for this BS to gey resolved.
http://www.thewebsiteisdown.com/

Not happy at all!! This is the busiest time for my site and it’s cut off right in the middle of it. Poor communication. Get this shit sorted NOW!!!!!!

March 23rd, 2012 at 3:17 pmPeter Kaizer Says:

Fix it or I am moving everything to Rackspace! You guys are getting on my nerves…..

Every other friday… everyting is busted!

I have been extremely satisfied with my NEW hosting provider (after the last couple hundred DH outages i switched) have had NO downtime and excellent support VIA the telephone (such a n ovel concept) and by email. I suggest you move as dreamhost has proven their inability to conduct a business. This must be a couple of teenagers trying to turn their hobby of web servers into money. DOWN WITH DREAMHOST!!!!!

ARVIXE ALL THE WAY!!!!

March 23rd, 2012 at 3:18 pmBarry From Eastenders Says:

My bum is itchy, what do you recommend?

You need a failover for your failover. Seriously, this is what redundancy is supposed to prevent. I find it funny, that it isn’t all of Dreamhost sites that are down too. Some of mine are, and some others I administer for clients, are not luckily. I think its time to invest in a better backup solution.

this always happens at the worst times. but i guess there never is a good time for this shit. can’t they just get it together. always giving these explanations when there’s really no excuse for it. you can’t run a business like this. other host companies don’t have these issues. it’s time they start accepting some blame and making some changes to prevent this shit from always happening. why have they not learned?

You guys are down all the time lately.. no way I can run a business on this server.. Time for something more expensive like Rack Space, although I’ve heard good things about Hostgator from friends

Honestly, this is ridiculous. In this month alone we’ve been forced to suffer from multiple cases of significant downtime – along with a multitude of other, smaller issues. I don’t know what kind of shenanigans are going on over there, but this is out of control. I’m tired of receive sub-par service. Working with your company used to be a pleasurable experience, now it’s just a headache. I’m going to be writing a letter to your support staff. If the response isn’t appropriate, I’m out of here. No time for these games.

Stuff happens. Appreciate your efforts, DHost.

We are back! Thank God, was getting tired of picking up the phone and pretending to talk everytime someone came in my office to complain.

You need a failover for your failover. Seriously, this is what redundancy is supposed to prevent. I find it funny, that it isn’t all of Dreamhost sites that are down too. Some of mine are, and some others I administer for clients, are not luckily. I think its time to invest in a better backup solution.

Ha! Just tried posting this and got a “502 Bad Gateway” from dreamhoststatus.com. maybe this will go through this time.

I have been extremely satisfied with my NEW hosting provider (after the last couple hundred DH outages i switched) have had NO downtime and excellent support VIA the telephone (such a n novel concept) and by email. I suggest you move as dreamhost has proven their inability to conduct a business. This must be a couple of teenagers trying to turn their hobby of web servers into money. DOWN WITH DREAMHOST!!!!!

ARVIXE ALL THE WAY!!!!

You guys should work as hard to prevent these issues as you say you do to fix them

I guess I’m surprised that a hosting site (or any datacenter for that matter) would only have only one network circuit. Even the junkiest DC I’ve ever worked in had something in place for failover with a different upstream network provider.. Is there a reason that there’s only the one?

It seems its down again, I cant even log into the dreamhost CP.

This is horrible. I am applying for jobs and people look at my online portfolio. My online portfolio is hosted by DreamHost! Argh.

Paul, going to HostGator is a big fuckup. You’ll see.

Ugh. This could not happen at a more inconvenient time. I’ve got to finish a project before the end of the business week and an essential document is attached to an email in my (unreachable) inbox. :(

BACK UP!
BACK UP!
BACK UP!
BACK UP!
BACK UP!BACK UP!
BACK UP!BACK UP!BACK UP!
VBACK UP!
BACK UP!BACK UP!
BACK UP!

a week ago I was recommended your hosting service to a new client. And right now he is asking me WHY the website demo is offline.

Please, fix that asap.

WHOIS Data:
Technical Contact:
DreamHost Web Hosting
New Dream Network, LLC.
PMB #257
417 Associated Rd.
Brea, CA 92821
US
+1.7147064182

Just in case anyone was wondering and wanted to leave a Voicemail for our illustrious Hosting Service (Not that they seem to listen to their customers)

That’s it. I’m taking my business and clientele off of Dreamhost. This happens all too often.

Dreamhost, how about you just let everyone use space on the dreamhoststatus.com server? That’s the only one that never seems to go down.

All my sites, and clients are down, not long time was a problem with hacked accounts. Should i start thinking to stop using dreamhost? No good!

What are the good alternatives for VPS?

COME ON…..

What I don’t understand is their web panel/system status/ etc is on line and my stuff ain’t. Hmmmmmm.

Yikes. Looks like everything’s down.

It scared me because I was doing some (rare) work in my WordPress admin, using a Theme Test Drive plugin. Thought I broke something!

Hope it’s fixed soon!

March 23rd, 2012 at 3:22 pmgreenfishfiles Says:

Oooh! Webmail, panel and site are coming up! Let’s see how long it lasts….

Been with DH since 2005 and in the last 6 months they have had more down time then all the years before combined.

It doesn’t matter how hard they work, this is hurting business. It is time to move on. In looking up reviews I found http://www.inmotionhosting.com/hostingplans.html to have some of the best reviews. They also have 24 hour support chat.

I will move to them and hope for better service.

Thanks for all your hard work dreamhost. But your aging hardware or just incredibly bad luck has cost me to much to stay.

Ok guys, this has been an off and on problem, but in a lot of ways still better that most. I have been using DreamHost for a while and will continue to do so for some time. But I also have a couple of sites at two other hosting companies that I have been trying out for a little while now.

Give them a try. They offer everything from shared hosting to fully dedicated servers, all at a fair price.

You can check them out at http://www.magnoliasweets.us/ and http://www.et-host.com/

I’ve been with DH for over a decade with several sites for clients hosted with them, but luckily I’m currently only hosting a single site on the shared servers. Luckily it’s only a personal site, but unluckily it’s a personal site which contains materials that need to be reviewed by potential employers while I search for full-time employment, and unluckily I just sent out an array of applications, and unluckily my email that I use for communicating with said potential employers is also hosted on DH. Up until the hack a few weeks back, I was a very happy customer. I’ve always liked their Panel user interface, I like their voting system, I like the services offered like QTSP support (though I no longer use this), WebDAV, etc., and I’ve enjoyed their casual approach in the newsletters while maintaining a professional uptime and communicating directly to customers when problems do arise. However, the past few weeks have been terrible with these massive failures of either email service, subsets of hosted sites, or the entire DH network in a complete blackout for hours on end during weekday business hours.

So what’s really going on, DH? I know you say you don’t read these comments, but I’m sure someone over there is skimming these to see how many potential customers are considering jumping ship. Did the hack just expose the weak structure that DH has been relying upon with appalling lack of redundancy? Did new senior management step in and ruin what was once a fantastic operation? So, which is it?

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

Seems it is weekly. No sites/email/panel, Dreamhost seem to be going down in flames!

I have been with DH for several years. However, the last few months they have given me little reason to stay with them.

“Update Mar 23rd, 3:10pm PST: We’re still working on restoring services as quickly as possible. Unfortunately we’re still experiencing network instability for which we apologize. We are aware that this is causing panel service interruption in addition to impacting other customer services, including web sites and e-mail. Our networking, system engineering and datacenter operations teams are working feverishly to resolve these issues.”

Hey. Dreamhost. This and the previous status update are exactly the sort of posting that ought to contain the key acronym: ETR. If you are not confident of your ability to resolve an issue, telling us what is being done isn’t so very important, and adverbs such as “feverishly” are neither impressive nor informative.

Yup, site just came back online. Do we all get discounts? Rebates?

Oh Shit,

IT WORKS DUDE! CHILL!

any one want more from dreamhost. Please u can move . send mail to backup@dreamhost.com get back your file and DB . brandon

I’m DREAMing of HOSTing my shiznit elsewhere.

Ron….i saw that 2 months ago….They are facing a serious problem in my opinion. And at the end, they will ask us to change password….

This is a great site http://www.uberflashgames.com when it comes back up

any one want move from dreamhost. Please u can move . send mail to backup@dreamhost.com get back your file and DB . brandon

March 23rd, 2012 at 3:24 pmBonnie L. Nadri Says:

No rudeness, guys, I know you’re working hard, but please consider me another 6 year customer who is beginning to wonder what has happened to the company I have come to rely on. Growing pains, perhaps? Understandable to an extent, but I cannot grow my own business if I cannot rely on yours; please review and revisit and if needed, re-fund your NOC for what you know it needs… redundancy, redundancy, redundancy, and another layer of fail-over protection.

Please.

my panel is up but not my sites and email

Unbelievably, still down. Can’t point our customers to our website, our blogs, can’t sync our source code control. I’ve paused all our paid search so we’re not paying to send people to a broken page. We’re finished with dreamhost.

Yay business…. moving users from DH to anywhere else…

If interested send an email to dreamhostmovers@gmail.com

I would have used a DH hosted email but unfortunately…

OK GUYS, ITS COMING BACK. DREAMHOST.COM AND PANEL.DREAMHOST.COM ARE UP, AND I JUST CHECK ONE OF MY SITES AND CAME UP.
NOW ITS NOT SHOWING MY SITES BUT ITS COMING BACK. PARTIAL RELEAF.

My website is down too. This is at least the THIRD major outage in the past few weeks. What is going on?? This is unacceptable and we should be compensated. Every time our service is down, we lose money. What exactly are we paying for here at Dreamhost???

Oh man, my daily average $.08 from adwords is gonna be cut in half :(

This is brutal :(

……..but I do love dreamhost.

my sites is down

Guys, WTH? I’ve been with you for, well, ever. I moved my stuff to private servers so crap like this wouldn’t happen.

Three hours and no update?

Downtime I can understand. Crap happens. But lack of updates is a management thing and is VERY preventable and VERY easy to fix.

After 10 years at Dreamhost, I am now forced to move my business.

No email. No sites up.

I can’t even get into FTP and MySQL to start to get all my sites and data.

NIGHTMAREHOST!!

FUCK!!!

Arghhhhhhhh,
why all this problem ?

March 23rd, 2012 at 3:26 pmGreg Hiscott Says:

Wow. These outages and glitches are in an increasing trend. This is not acceptable.

It’s back! Thanks Dreamhost your the best. I knew you guys would work out the bugs. Much love
XXXOOXOXOXOXOX

OMG, like, my totally important/vital site which brings me hundreds of $$$ per hour (mom’s blog of cat pictures) is down and I’m dead in the water losing money left and right.

or

This is the 4th time this year…that I’ve been totally done with DreamHost.

I say, don’t host a vital site on anything less than a dedicated box…if you don’t think enough of it to do that, it isn’t that “vital”. Also, if you’re going to leave, stop wasting time and GO DO IT. No one wants to hear your whinging and moaning, just leave. The rest of us will enjoy the added capacity now that your poorly coded cat photo blog with bogged-down google maps api calls is hosted somewhere else.

SSH connection available!!!

My site is still down.

Last week I moved my PS hosting to InMotion (great service by the way). I guess now I have to move my shared hosting too. :(

Not mail, not webmail, no panel no site i want a die help pleaseeeeeeee

this would suck if my livelihood depended on it…better release your lawyers from their kennels, DH

Another outage? Most of a business day without email? Dreamhost, get your act together, or we’re going elsewhere. You used to be so reliable, but we’ve had FOUR significant problems in the last couple of months, all of which have cost us a lot in resolution time and productivity.

But just came back.

For Accounts payment your sistem never fails but all your services fails every week

IT JUST ANOTHER DOWNTIME…..

Ron please stop using this as a marketing tactic to move people away from DH.

Why would you have DH email if your job is to move people away from it anyway?

mysql database still down

This really couldn’t have come at a worse time, just as some sites are publishing about my products this happens and people who visit my project page got no pictures…. :( :( :(
Will not be able to rely on dreamhost again soon, this was very bad timing… Why so many problems?

Thanks that you fixed it though.

Let’s start a pool to bet on the exact time the system starts functioning…$5 each….

My entry is 3:57 pm Pacific Daylight Time….Sunday, March 25!

Next….

March 23rd, 2012 at 3:28 pmBarry From Eastenders Says:

Blame it on the boogie, that fucking boogie

thank god. my site working good.

http://sahratarabia.com

latest World events & parties details update here.

I put a few extra beers in the fridge for the Dreamhost support folks. They’re gonna earn ‘em today.

Is coming back up!! It’s ALIVE!! Meckaleckahideehideeho

I just paid for dreamhost yesterday, and part of the reason I went with them was for a 100% uptime guarantee! So less than 24 hours later, and I can’t access my work in progress website, panel, or upload anything by FTP. And the “100%” is only if you open a ticket to begin recording the credit. So, this has been going on for what, 4-5 hours now, yet I’ve only just discovered it because I am back home. Looks like not much of a credit for me.

Still though, I am excited to have webhosting and my first website up.

I can’t believe in this horrible service, I will change my domains from here.

“Our networking, system engineering and datacenter operations teams are working feverishly to resolve these issues.”

*gets mental image of a band of chimpanzees with spanners smashing everything up in a server room*

@OMG — the sound of your “whinging” is resplendent.

OMG you are right. Fuck it – I am out of here.
My mom is pissed and so are her cats.

I’m done with you guys.

I am past Angry, this is just Funny now!!!
Website,FTP,Mail,Panel everything DOWN!!!!

Thanks DH!

Dreamhost sux donkey balls. WTF, whatever happen to 99.9% up time. It’s more like 9.99%!

This is happening way too often, we are a bussiness. The cost savings of Dreamhost is no longer cost effective with the lost productivity and customer contact we have when down.

I’ve been with DH since 2005. This is starting to get seriously annoying.

My sites are down, I can’t get my email – and I’m waiting for some very important correspondence. My renewal comes up this year, and right now I’m debating pulling my hosting. And I’m starting to regret all the people I’ve referred here.

Fix it and don’t let this happen before I renew hosting in Sep. Some prolonged up time would be nice. PLEASE. Until this year it was such awesome service.

March 23rd, 2012 at 3:30 pmBarry From Eastenders Says:

I couldn’t have come at a worse time

booo.. switching hosts is such a PITA, but I guess it’s that time. Every time i ask about loading time, dreamhost blames it on wordpress. But when they reset the servers – like that intermittent time around 1 today when everything went back up – the sites load blazingly flawless. I’m starting to think the mysql servers are overloaded and work slowly until they don’t at all and it actually has nothing to do with “bad worpress scripts”.

someone’s getting fired….

My sites hosted on steelers.dreamhost.com were down a moment ago, just now up… not sure if it’s intermittent.

Who wants to party?

Stephen here from DreamHost. We’ve isolated the cause of the instability, and currently Jeff from Amalgamated Vacuums is venturing deep into the core of the datacenter, dry-vaccing (that’s tech-speak for Dry Vacuuming btw) the spilled slurpee out of the main servers. After that, we’ll do a couple core reboots just to make sure that data is flowing freely. Hang tight, people!

What can I say, Fridays are Slurpee Day at Dreamhost HQ.

Bye, bye Dreamhost. It hasn’t been fun…

@OMG You are an idiot.

@KT indeed. I’ve been with them for 7-8 years….last 2 years have been super problematic.

jimmy … ASCII FAIL!

This is insanity! I’m trying to run a business here, and I’m now terrified of what’s going to be in my inbox once service is restored. This is FAR too long of a downtime to be considered acceptable, YOU ARE COSTING ME MONEY!!!

Well this is inconvenient.

Get out from dream asap.
Some day all the services will never come back online.
I’ll move my last one next month!

Hey, we’re back up… THANKS DREAMHOST YOU GUYS ARE EFFING AWESOME!!!

Ottawa calling. Email and webmail has been down now for 4.5 hours here. Always occurs of course in the middle of a busy day with some major transactions happening. PLEASE get the system back up. Thanks

March 23rd, 2012 at 3:33 pmLongtime DH Customer Says:

Been here since 2001 and looking hard at why I should stay!!!!!

I can spend my money else where for a better peace of mind……

I pay you to manage this and let me sleep at night…….now it’s just NIGHTMARES!!!

http://dreamhost.com/web-hosting/

The DreamHost 100% Uptime Guarantee
DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

I also have a dog named Trudy.

I’m quoting @steven because I think he is dead on:

“booo.. switching hosts is such a PITA, but I guess it’s that time. Every time i ask about loading time, dreamhost blames it on wordpress. But when they reset the servers – like that intermittent time around 1 today when everything went back up – the sites load blazingly flawless. I’m starting to think the mysql servers are overloaded and work slowly until they don’t at all and it actually has nothing to do with “bad worpress scripts”.”

I think the proliferation of blogs has outstripped Dreamhost’s ability to serve up mysql.

tumblr is looking awful good right now.

Haven’t been able to reach my mailboxes for 9 hours, and about 200 of my websites are down. Not amused.

You know, when my website goes down, I don’t complain. When my SSH access (along with my Git repositories) goes down, I grin and bear it. But email as well? Guys, you’re killing me! It’s nice to have a peaceful afternoon on Friday, but another few bouts like this and me (and my 30 odd clients) are going to need to be shopping around. Email’s critical (and yes, I know I can do Google Apps, I’m not sure I trust them, thanks though).

Btw, first time post/complaint ever on here. Hopefully it’s the latest or it may be the last. :-/

I’m not angry.
I’m not angry..
I’m not angry…

I’M NOT ANGRY!!!!!!!! ARRRRGH!

@Cicada my site just went back up. All I can hope is that I haven’t lost email. Flood of messages incoming…

Please get this back up! We are a really small biz and our business is completely dependent on our customers being able to schedule classes from our site, which they typically do the day before. Because we have been down almost all day we have only one sign up for our class tomorrow morning. I’m going to have pissed customers and this directly hurts our sales.

“It Friday, Dreamhost must be down”. Definitely going to start looking. It’s such a pain in the ass, though.

March 23rd, 2012 at 3:36 pmLaughingPurring Kitty Says:

@Stephen Ballswell
Lololololololoolol

mail down for me. As others trying to run a business here.

Try to open a ticket on a dead site to get your credit.

Mail is up. site not quite yet. Gaining ground… kind of scary though not showing any new messages… Not likely on a Friday afternoon

It’s a pain in the ass to look, and since these are my dev servers, I don’t really care – just a little lost productivity time. But, I am certainly not going to recommend Dreamhost to anyone. They will lose a lot of business just in referrals.

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

I’ve been a DH customer for maybe 6-7 years now, and I still love it. Whenever I have a problem, their support is friendly and helpful, and when they do have problems like this, they’re up front about it, which is more than I could say for other hosts I’ve been with. I honestly have had very few problems with them in my many years with them, so I’m not fully getting all the anger and hate. I don’t rely on my websites for business though, so maybe that’s why.

HOW UNPROFESSIONAL! Your system should have enough redundancy built in so that this NEVER happens. DreamHost just screwed up my mass communications.

Many of my affiliates are getting “a little” angry that they can not communicate with the outside world due to prolonged e-mail downtime. An hour or two here and there during the past decade has been alright, but this time we’re starting to loose money over here in Minnesota.

Its not working ….

This is a real bummer. I sent formal emails last night inviting important people to look over an article I wrote on my Blog and all I have today is emails from them sayin that they CAN’T ACCESS THE WEBSITE! Makes me look like a complete idiot. This is the third time this has happened in a major way this year. Years ago DreamHost was reliable, now it’s best only for a “relatively stable” development environment. I am migrating all my critical clients and sites to websynthesis(dot)com as of Today. It’s for Genesis powered WordPress websites ONLY. Perfect.

my site has been either down or replicating dialup speeds for the last 40 hours. now email not working. anyone on here have other hosting co. recommendations? this is brutal!!

My email and websites are back up.

I run a business and this is unacceptable. Your services have gotten worse over time. I have been with dreamhost since 1998, but will be leaving now.

This is simply unacceptable. I move to a much higher cost VPS just over a week ago, and there have been problems every single day. And now this. This is costing me money!

My sites still remain down, and we’re going on four hours now. I’ve also been with Dreamhost forever, but I’m “this” close to switching everything over somewhere else. Which, of course, I can’t do until I can actually access my sites to backup all my databases.

Ridiculous.

Wow … my little rant must have jared something loose!

Lots of downtime recently. This one so soon after the forced password changes too after your recent oversight of an unencrypted password file. Dropping the ball this year after so many years of reliable service. Might need to entice customers to stay after this. Not worth the price premium.

This is happening way too often!! We lose money every time dreamhost loses email. We will be sending you an invoice for damages. This has been a regular event since email went down for an entire week in Sept. at a cost of approx. $65,000.00 in lost revenue!!!!!!!!!!!!!!!!!!!!!!!!!! We lose far more than you guarantee.

Nightmarehost is back again

Remind me to get Brandon laid. My sites are back.

Services are back, Where are ALL my emails from 11:00 am PST until now? Fix this!

Email might be back up. It connects via pop, but no messages, and the test message sent hasn’t gone through.

No messages for the last 4 hours.

Children go outside and play. If this outage is affecting you to the point of costing you money, you’re doing it all wrong.

From http://dreamhost.com/web-hosting:

‘Our 100% uptime guarantee means we take uptime seriously.’

…and…

‘DreamHost is committed to customer satisfaction and every account is backed by a 97-day money-back guarantee.’

So, logically, this downtime being impossible, we’re starting over on the ninety-seven days? Whenever the account actually starts up again? Right?

Back again – let’s hope it sticks this time!

HOORAY, dreamhost.com is up right now. Everything back to normal for good……

Man I hope it sticks this time. It’s been TWICE in less than a month

hopefully everything is good now.

GODOMMOT FRONK

Back up and running. Thanks guys!

March 23rd, 2012 at 3:46 pmPaulo Figueiredo Says:

Working again… tnks for the downtime ;)

mail is working but where is all the email for the past three hours??

Who else believes that DH should give back an entire month as as credit?

March 23rd, 2012 at 3:47 pmwhat in the bejesus Says:

Stephen Ballswell Says:

Stephen here from DreamHost. We’ve isolated the cause of the instability, and currently Jeff from Amalgamated Vacuums is venturing deep into the core of the datacenter, dry-vaccing (that’s tech-speak for Dry Vacuuming btw) the spilled slurpee out of the main servers. After that, we’ll do a couple core reboots just to make sure that data is flowing freely. Hang tight, people!

What can I say, Fridays are Slurpee Day at Dreamhost HQ.

^^^^

I hope this is a joke. A couple of core reboots to make sure data is flowing freely? You’ve got to be kidding.

Please I need my emails server !!!!!!! I lost all my email since this morning

My website is still down?

What is going on? We have visitors that are complaining. I won’t think twice to host my site else where.

Some emails that were deleted 12 hours ago are reappearing in my inbox, but the only current email is from about 15 minutes ago. I wonder…is the afternoon’s email all lost, or will it dribble in over time (along with more of the spam I deleted this morning)?

Well the good news, my site is back up and emails are functioning.

The bad news, any emails sent to me have been lost, just dissapeared off the system, with no delivery fail sent back to the senders…..wonderful…

Thanks!

Well, my email seems to be back up but important mail messages that people are expecting me to respond to have not come through. Zero messages after several hours of downtime. I hope they are not lost for good.

Wow, you guys quoting that 100% guarantee seem to be missing the part wherein they’ll credit you for downtime like this. Yeesh.

And my beloved blog is back up! Yaaay!

Also wondering whether email received during outage is lost.

both sites on seperate servers back up. How did they consider this a small subset of users when the entire panel went offline?

“The bad news, any emails sent to me have been lost, just dissapeared off the system, with no delivery fail sent back to the senders…..wonderful…”

I didn’t think to send myself any test emails this time. I did last time there was a major blackout on DH, and I think I did receive a bounced reply back then….

Can anyone else verify?!!

.

Wow ppl, get a clue. Just because the routers are routing doesn’t mean that all the backed up emails *instantly* appear. There is an email queue that must be processed once everything is verified as working for everyone on their end. Quit being so selfish and let everyone else use this thread to bitch about their sites being down before you go and hog it with you ‘now where is my email’ stuff. Sheesh, what are you, American?

my sites are bake and I’m outta here

Wow all of Asmara down. That would be 8 websites I have with you guys. This is getting to be a regular thing.

Please fix your data server.

“Well, my email seems to be back up but important mail messages that people are expecting me to respond to have not come through. Zero messages after several hours of downtime. I hope they are not lost for good.”

Well … that is … not good….

Hmmm… some of my sites are back again – bout all from the same account. Others are still NTBS

Would like to know what sort of credit users will get for this? My clients are not impressed.

We’re back up! The world has been saved! Thank you DH and Brandon for your updates and resolving the problem!!

email still down for me

Looks like I am back too. http://Syrito.com

I AM SWITCHING TO: WebSynthesis Hosting: http://www.WebSynthesis.com

I wish I was an Affiliate but I am not. Tell ‘em Syris sent you from Syrito.com
THEY WILL WELCOME ALL COMERS WITH RELIABLE, PROFESSIONAL, BUSINESS, GENESIS POWERED WORDPRESS WEB HOSTING

Simon Says it’s time to move to another hosting company.

all services working!!

Brandon, what about incoming email???

I would like a 4 hour refund !!! I don’t know how long the site was down- judging by your status announcements i’m estimating four.. Your up time guarantee is false advertising..

Class Action Lawsuit

This thread is crazy long. A lot of people must be affected. Not very many of them are going to get any refund! Did you file a support ticket?

Not that it matters, the refund has a max of “up to 10% of your next pre-paid hosting renewal fee”, which apparently would mean less than $1 for most people.

LOL, I still can’t get over that 10%. Even if the server is down 100% of the month, they only give you 10% of your money back!

March 23rd, 2012 at 4:04 pmPast customer Says:

I was a DH customer for many years, but a couple of months ago I noticed service was really starting to slide and moved all my domains and sites to another hosting company. Fees were about the same as DH, though I had to pay to update all the domain names when moving them.

New host is great, really. No downtime, very fast mail servers, fast, helpful chat available 24/7, free phone if needed. I wish I had moved a few months before and saved some hassles.

Look, folks, there are much better choices out there, and while it’s a pain to move everything, once it’s done, you’re free of all this crap from DreamHost.

“You really should change your company name from DreamHost to NightmareHost.” to whomever posted this, that’s exactly what one of my co workers said when I was asked why the e-mail wasn’t working and announced over the PA “E-mail is not working nor is our website because Dreamhost is down again. I’m soooooooooooooooooooo glad I don’t have to deal with the website here at work anymore. No more calls of “Why is our website down?!?!” “Let’s see…oh Dreamhost is down again” Now I just have to tell them why e-mail isn’t working so use the gmail accounts I setup for you because who knows how long it will be out.

BTW; e-mail is still not fully working. I’m getting bad account/password errors most the time.

We need some sort of compensation for all this heartache you put us through every time

Compensation NOW

and not all cheap hosting is down this much. I have my own websites on another host that starts with Mo and have had them for several years and have only seen 1 time that I couldn’t get to my sites and that was only a couple mins. I’m not selling for another host which is why I didn’t post their whole name, but no this doesn’t happen all the time everywhere else.

The contents of my inbox are gone. Not happy.

Please next time when the server is down temporarily, would you just let us know??? i wasn’t even aware until my client told me!! imagine how bad that is…

From my end, I’d be happy for DH to increase their charges if I got more reliability in return. I seem to be getting my ass kicked on monthly basis now by my clients whose website are down….

What a terrible service offered by DH. We and all of our customers are out of here.

Thanks for f**king nothing dreamhost.

THANKS GUYS!!!

The Better Business Bureau has a big fat F for the umbrella company of Dreamhost – New Dream Network. It seems this company likes to dream of outstanding service

The entire contents of my “old messages” IMAP folder are gone — 20,000 emails dating back several years. These NEED to be restored… that’s a huge, huge problem for me if not.

Websites, panel and zovoort are back for me.
Thanks … hope this will not happened to much!

How could someone open a support ticket for the panel being down? please someone help me understand that statement?

Signed the petition…passing it on..

__,
.-‘ / ,——–,
.’ / /`. ,——–,’ |
| / / | | _ | |
| \__/ | | .’ `. | |
`. .’ | `._.’ | |
`. .’ | |,’,-,-,-,-,-,-,-,-,-,-,-,-,-,-==,
| ][ | |__________\——————————==’
| ][ |
| ][ |
| ][ | .—-. .—.
| ][ | ‘—, `.____________________________.’ _ `.
| ][ | ) ____________________________ :
| ][ | .—‘ .’ `. .’
| ][ | `—-‘ `—‘
| ][ |
.’ __ `.
| / \ | jgs
| \__/ |
`. .’
`—-‘

___ _ _ _ ____
| | |\/| |___
… | | | | |___
,’ ‘. ___ ____
; ; | | |
| | | |__|
| | _ _ _ ____ ____ _ _
| _, ,_ | | | | | | |__/ |_/
‘´ | | `’ |_|_| |__| | \ | \_
| | _ _
| | .|_|_,’ | ,–.
| | | | : |
| | `|`’`-._| : |
| | | | : |
| | | | : |
| | | | .——-. : |
| | | | ,| _O | : |
| | | | ||_,’ | : |
| | | | `| | ;…:fsr
| | | | | | /.–. \
| | |` | | | |’–‘ |
‘-‘ `–‘ ‘——-‘ ‘—–‘

not receiving emails, pop and webmail, just sending ok.

If you can find something as complete and easy as dreamhost … for a few dollars by month … please advise.
If you want more 99.9% … think redundancy.
But I agree that this year seems to be haunted by some flaws.
But I love the Panel and the word ‘unlimited’ nearly everywhere :)

Still having mail problems!!!!

I’m back up!!!

and so are my sites

literally

throbbing

with liquid

mucosal

joy

unleash
your

inner
bi-gendered

emosexual
love beast

and join me

I AM SWITCHING TO: WebSynthesis Hosting: http://www.WebSynthesis.com

I wish I was an Affiliate but I am not. Tell ‘em Syris sent you from Syrito.com
THEY WILL WELCOME ALL COMERS WITH RELIABLE, PROFESSIONAL, BUSINESS, GENESIS POWERED WORDPRESS WEB HOSTING

DH is back, let’s get naked!!!

March 23rd, 2012 at 4:38 pmTime_to_change Says:

I’m switching this weekend, Dreamhost will no longer cut it for me. I need reliability and support and I’m willing to pay for it.

Pretty sure I’m going to go with http://www.liquidweb.com/, it seems like they have their priorities straight.

I don’t understand how our website could be down for 4 hours when we have VPS and VPSMySQL. We pay extra money to prevent things like this. Really curious as to what DH response will be when all is well again. We definitely want a credit.

So regarding emails sent to DH servers during the outage … can anyone confirm if those eventually re-propagated to their accounts, or whether those messages were sent back to the senders with an error message, or whether those messages were just lost in the ether?

Tired of downtime? Sick of excuses? Me too!

SWITCH WITH ME TO: WebSynthesis Hosting: http://www.WebSynthesis.com

Tell ‘em SYRIS sent you from Syrito.com, they will set you up for real. Remember who sent you.

THEY WELCOME ALL COMERS WITH 100% RELIABLE, PROFESSIONAL, BUSINESS, GENESIS POWERED, WORDPRESS WEB HOSTING

Hallelujah!!!

Well, from the most recent couple of updates by Brandon; “All e-mail affected by this issue will be sent, however may be delayed for 4-6 hours.”

I just love dreamhost and all the good service, this is nothing for all the great things that i do with 8.95/mo!

¡For me this problems are nothing, im happy with Dremhost and i love it!

And really… if you loose a looooot of money for be offline… buy a dedicate server… really….

Happy user from Colombia!

My admin panel in Worpress is not there anymore. There’s no more wp-admin/ folder and it redirects to 404 error.

Are you tired of downtime? Are you ick of excuses? Me too!

SWITCH WEBHOSTS WITH ME TO MAKE A STATEMENT: WebSynthesis Hosting: http://www.WebSynthesis.com

Tell ‘em SYRIS sent you from Syrito.com, they will set you up for real. Remember who sent you.

THIS IS THE BEST, MOST RELIABLE WORDPRESS WEB HOSTING ON THE PLANET!

100% RELIABLE, PROFESSIONAL, BUSINESS, GENESIS POWERED, WORDPRESS WEB HOSTING, ONLY FOR PROFESSIONALS!

Last email I received was 10:44 AM EDT. It’s now 8:00 PM EDT. More than 9 hours. And I don’t even use Dreamhost for email. It’s just redirected so I can use my domain names. My account is set to renew in April. I have another account, that expires in July, with a company that has given me good service since 1996. Hmmm. I wonder what I should do? Or not do? I think I know…

March 23rd, 2012 at 5:24 pmMarcelo JImenez Says:

Dreamhost rules!!!

If there is problem, nobody else will tell you what’s really happening.

Regards

Well, looking at the webstynthesis.com plan pricings just reminded me why I use dreamhost. ouch. For $97 a month you get to host TWO WHOLE SITES!

March 23rd, 2012 at 5:28 pmMarcelo JImenez Says:

@Past customer

So, say what hosting company you moved on….

Yes, it was irritating and frustrating to be unable to do site maintenance this afternoon in the time I had cleared specifically for that purpose. And, yes, I think the updates didn’t acknowledge the size of the problem and came across as minimizing the number of people/sites impacted.

But…

For $8.95 a month I am able to host 12 live sites, plus a couple for learning and experimentation, with pretty few restrictions. Support – which I have only needed infrequently – has been much more than I would expect for $110/year.

And the user community here is awesome.

If I were hosting a big dollar commercial site, I wouldn’t be doing it for $8.95 a month. (Or, if I did, I would deserve the consequences.) The support and responsiveness and reliability here are head and shoulders higher than what I have experienced in the medical professional areas I work.

(@SYRIS: anyone who has to say the same thing 3 times, uses CAPS so liberally, and claims 100% reliable…makes me think of the guy who wants to sell me the best vacuum cleaner in the world that picks up everything, never breaks, for only $19.99 down an $9.99/month for 20 years. If I buy now and pay cash, he’ll through in the knives that never get dull…)

Look at all these jackasses trying to bump their referral bonuses on their hosts. Comedy.

I found a web hosting company with a special deal at $3.49 or $4.99 a month for a year, and regular price is just twice that. Still less than Dreamhost:
http://order.1and1.com/Hosting?ac=OM.US.US430K06689T5506a&ovm_kp=hosting&ovm_sc=awareness&gclid=CKv309Ok_q4CFYbe4AodZg_v1Q

Just search “web hosting”. That’s how I found Dreamhost.

All services are NOT RESTORED — I’m still down…. this beyond sucks on so many levels.

I have been a happy customer of Dreamhost for more than a decade and this is the first time they have had a chronic issue that has crippled our ability to service our clients and unfortunately has deeply effected our confidence in dreamhost. Honestly, I am worried that the recent acquisition of Dreamhost by Endurance International Group (last spring) was the tipping point………something has changed fundamentally. Either it is poor planning or poor execution……regardless it seems that this acquisition may not have been an upgrade but rather a downgrade. Dreamhost founders and employees where made millionaires overnight and obviously the key ones who had the passion did not stay with the organization or no longer care to maintain the high quality service, support, security and stability…..What a shame!
Where is my Host of old…..where is my Dream Host?

@Todd I agree that our confidence has been shattered. However Dreamhost was not aquired by Endurance (that was an April Fools joke on their Blog).

The problems are a result of Dreamhost having 2 Datacenters with VPS accounts and MySQL services split between the two datacenters. So your VPS host machine may be in Datacenter “One” and your MySQL host machine in Datacenter “Two”. This design is a huge problem because if one Datacenter loses it’s link it brings down sites hosted on the other.

I have been a Dreamhost customer for along time and all of the problems with instability, outages and downtime etc. started when they opened the second datacenter.

That coupled with VPS host machine RAID Array failures due to cheap-ass hardware and poor fault tolerance has led to a really piss poor experience for us Dreamhost customers as of late. Who ever has heard of so many Raid Card failures????

Someone help me please! We have a school website that uses the WebCalendar. Yesterday, a mobile user was redirected to a porn site. Today, the same thing. Not all the time, just seemingly randomly. Another user (I believe PC) was also redirected, but his anti-virus caught it before the graphics came up. I have emailed Dreamhost and am waiting a response, but this site is used by many young children and I need immediate help. I have removed all links to the calendar at this time, but the users depend on it for daily information. I am not real computer savvy and could use all the help possible. Thanks in advance. My apologies if this is not an appropriate area to vent.

Thank you guys for your great work getting this resolved so quickly.

This is happening at a much too frequent rate. Moving our cloud to rackspace. Anyone that says “thanks for getting this resolved so quickly” Is providing a terribly low level of service to their clients. Our clients webmail was down for over 4 hours, and the same thing happened last month.

Yes, they have had 2 major outages in 2 months. Before that, almost nothing that affected me for the best part of 3 years. I have seen much worse at places that charge more.

The problems are a result of Dreamhost having 2 Datacenters with VPS accounts and MySQL services split between the two datacenters. So your VPS host machine may be in Datacenter “One” and your MySQL host machine in Datacenter “Two”. This design is a huge problem because if one Datacenter loses it’s link it brings down sites hosted on the other.

I have been a Dreamhost customer for along time and all of the problems with instability, outages and downtime etc. started when they opened the second datacenter.

Well, that’s interesting.

12 hours down and counting….

Finally got my emails at 1:15 AM EDT. 15 hours downtime.

This is NOT good. Any bets on how long before another outage?

@Julie, It sounds like your site’s php files are injected with malicious code (either at the top of the file or at the bottom). You need to delete all the files of the site and upload back up (or download the files and clean them one by one).

March 24th, 2012 at 12:01 amAnother Random Emo Says:

Hey Emosexual, let’s get it on sweety

Good gawd the amount of fcuktards posting links to other hosts to try to get referral bonuses is insane. Perma IP ban every single one of those motherfcukers.

Ben Jarvis Says: anyone like Fatcow?

Yes – charcoal grilled filet mignon medium rare with mushrooms, plus some garlic mashed potato & gravy on the side, and a small crisp salad. Awesome.

Websites back up, email started arriving. Still missing 4 hours of email from every single email account. Trying to contact Dreamhost to submit a support ticket — they can’t be reached!! Panel is down again, seriously?? Aargh. Could not be more frustrating. Every bit of time I have to spend on this kind of crap costs me money; I also will be moving to a new host.

Dreamhost panel is down at time of writing. Discovered now after this issue was resolved that my site was hacked. by rr.nu. I recon I can get at least 50% of the costs of cleaning my site from Dreamhost?

yep i can concur with seraphiel

Hi

The Dreamhost Panel is down,

A new test for my patience, please fix the issue quickly…

Hi, my panel is not reachable and I just wonder hoe I am supposed to place a ticket regarding this …???
/Anders

March 24th, 2012 at 6:14 amXfactornorton Says:

Anyone else who is fed up with the terrible customer service of Dream host >>>>My recommendation is too dump Dream host and get a hosting service with someone that has a clue!

My sites were just offline again for a good half hour.

WTF dreamhost?

This is getting to be beyond a joke. DH has fallen to a new low today.

We’re finally biting the bullet and moving all our own sites elsewhere and recommending that all our nonprofit clients do the same.

We look bad when DH — yet again — fails to deliver.

DH’s time has been and gone. You blew it.

What a mess, guys!

Marco: Lol @ spending £5/month for a business olol

Why can’t I get a response from my ticket I submitted at 8am Eastern time??? We aren’t getting emails from our lead contact form which is costing us $$$$$ and Dreamhost can’t even give me a response back in over 8 hours! Still waiting…………

hi all-

what other services out there are decent? fat cow? others? i know speakeasy is as bad as dreamhost. please advise. this is now comical. thanks.

This was the straw that broke the camel’s back for me. I’ve just signed up for VPS hosting with InMotion Hosting. Am currently moving my business to them. They’ve been really helpful and my site is already running noticeably quicker.

So long Dreamhost.

The “you get what you pay for” argument doesn’t work here. Sure $10 a month is cheap, but this was a network-wide outage. It knocked out the people paying a lot more for VPS and dedicated servers also.

If I was paying $250 a month for a dedicated server I’d be very, very angry about all these long network outages.

Ours seemed to be back running much faster in fact by 5pm Friday PDT. When something we depend on fails, it sucks. But then nothing is infallible when dealing with the internet. DH, you have had your share recently. Let’s hope it is random since the alternative is not good for either customers or company.

I had just finished moving my last clients off of DreamHost when this happened. Completely unaffected by it as a result. I urge others to do the same for your peace of mind.

Everything still down?

Still down here….

My site is down and I cannot access anything – that doesn’t sound like all is well to me! When will this be resolved?

My site isn’t on this list, and it was down for most of last night, but no mention of this from Dreamhost.

What’s going on? It seems like you got a new CEO and decided to cut cots and cut quality of service.

I pay $50/month for VPS hosting. That’s not a lot of money, but it’s not money paid for nothing either. Are you serious about offering a service worth that amount of money, or are you just banking on your name and hoping no one will notice?

Seems after your “Down time” everything is not ok. Looks to be our Panel and account has been hacked again. All instances happening during DreamHost downtime. Ridiculous! I can’t wait to get away from this company.

My site has been down for two weeks. It’s been noticable and effecting my ability to do business.

for a different domain and email we use RackSpace.

If it is any consolation to anyone, they have been having issues today.

In a perfect world there would be no downtime, but “stuff” happpens. I am looking forward to hearing of DH enhancements to avoid last fridays fiasco.

from our support team:
For the 3rd time today RackSpace is reporting system disruptions. We will keep you informed. Our email has been hit or miss with sending today….

Outages again. 99% uptime must actually mean 99% intermittent uptime. with DH “stuff” happens multiple times per week. 100s of outages and hardware failures per year.

Anyone else still notice their e-mail is acting flaky? Sometimes it will download new mail, other times it’s failing and getting a bad account / password error.

I see above me that kirk is also experiencing what sounds like the same thing

I left Dreamhost after their problems earlier this year… but I ended up coming back. Say what you will about their recent problems, but for $10/mo the service is a heck of a lot better than most alternatives. If you’re running a VPS I can understand being angry. But I paid $20/mo at another company where a support ticket has sat without any response for three weeks! I really missed the DH panel, too. And at least DH is up front about their problems — they even allow us to flame them in comments on this blog!

Dreamhost definitely has a lot of work to do, as their reliability has suffered greatly this year, but if you are someone who is just hosting a few small sites for $10/mo, it’s not worth the effort to migrate it all somewhere else only to have to deal with a new set of problems.

Yes, same issues here – email issues since this morning, password error/server unavailable. My server is Albali. Webmail doesn’t work, which is the litmus test.

Dear Dreamhost, please let us know what’s going on. Need information + status, please.

Anyone using Amazon EC2 services?

DOES DREAMHOST NOW TAKE DAYS TO RESPOND TO SUPPORT REQUESTS NOW?! MY SITE HASN’T WORKED PROPERLY IN MONTHS, ONE DAY WE’RE TOLD IT’S DUE TO US USING TO MUCH MEMORY (WE AREN’T) THEN WE’RE TOLD IT’S DUE TO OTHER USERS CREATING A HIGH LOAD ON THEIR SERVERS, THEN WE’RE TOLD IT’S A CONNECTION ISSUE, OUR SITE NEVER WORKS PROPERLY ANYMORE (IT’S SLOW/IT GOES DOWN) AND DREAMHOST WON’T GET TO THE BOTTOM OF IT, WE’VE BEEN WITH YOU FOR YEARS BUT HAVE TO LEAVE IF YOU CAN STRAIGHTEN THINGS OUT, PLEASE PLEASE PLEASE GET BACK TO US SOMEONE IN SUPPORT. JOSE, BRIAN, ANYONE CAN YOU HELP US????????????????????!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Two days having VPS down on homie-vserver225! Two days! 48 hours! What sort of VPS is that? Would you sign for hosting service if someone tells you that downtime will be few days in a row?! Support does not exist. Support tickets are sitting there for days. If you have such luxury and can to move your site somewhere else today (having current copy of your DB) just do it. Do not wait until it’s too late.

Can I add that I am DH customer for 8 years…

My last support ticket with Dreamhost was open for 23 hours before I got a response. Their response was sorry for the delay responding but we don’t see any problems.

Meanwhile any of my database powered sites were slower than 1 legged dog on tranquilizers for the duration.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver194′ moving to new hardware and restoring from backups (RESOLVED)
8:01 pm

Web VPS host server ‘homie-vserver194′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 20th, 2012 at 8:01 pm PST) by Seohee

Due to issues with the hard drive on the web VPS host machine ‘homie-vserver194′, the /home partition has become RO (read-only). We are preparing the new hardware now for the move, and will be restoring all data from our backup server to the new hardware once the move is in progress. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will update this post as soon as the new hardware is setup and the move has begun. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver194′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 29th, 2:35pm PDT: This restore is now complete. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Oscar

Update Mar 29th, 8:04pm PDT:The last and largest VPS is still restoring from backups. The server seems to be working well in the meantime. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Seohee

Update Mar 29th, 10:54AM PDT:The last and largest VPS is still restoring from backups. The server seems to be working well in the meantime. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Justin

Update Mar 28th, 10:19PM PDT: This restore is nearly done and  our admin team is checking to make sure all final rsyncs have been completed. We will update this post again once the restore finishes. All VPS guests are up at this time as well. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Seohee

Update Mar 28th, 5:00am PDT: This restore is almost done and  our admin team is checking to make sure all final rsyncs have been completed. We will update this post again once the restore finishes. All VPS guests are up at this time as well. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Patrick

Update Mar 27th, 10:00pm PDT: This restore is almost done and  our admin team is checking to make sure all final rsyncs have been completed. We will update this post again once the restore finishes. All VPS guests are up at this time as well. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Dan J

Update Mar 27th, 12:35pm PDT: This restore is almost done and  our admin team is checking to make sure all final rsyncs have been completed. We will update this post again once the restore finishes. All VPS guests are up at this time as well. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Oscar

Update Mar 27th, 2:14am PDT: This restore is still on track and all VPS guests are online at this time. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! -Patrick

Update Mar 26th, 8:20pm PDT: The restore continues well and all VPS guests are up and running fine.  If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. –Dan J

Update Mar 25th, 5:20am PDT: The restore continues well continues to execute and all VPS services are reporting online. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support for help. – JJ G

Update Mar 24th, 9:30pm PDT: The restore continues well and all VPS guests are up and running fine. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Oscar

Update Mar 24th, 5:05am PDT: The restore continues well and all VPS guests are up and running fine. The load is looking better than is was so hopefully sites are loading faster as well. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Patrick

Update Mar 23rd, 9:03pm PDT: The restore continues well and all VPS guests are up and running fine. The load is looking better than is was so hopefully sites are loading faster as well. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Dan J

Update Mar 23rd, 9:45am PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Oscar

Update Mar 23rd, 12:45am PDT: This restore process is continuing without issue. There All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! -JJ G

Update Mar 23rd, 12:45am PDT: This restore process is continuing without issue. All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! -JJ G

Update Mar 22nd, 10:30am PDT: This restore is progressing without any issues! All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! -Justin

Update Mar 22nd, 4:00am PDT: This restore is progressing without any issues! All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! -Patrick

Update Mar 21st, 6:13pm PDT: All VPS guest machines are up and resolving without any issues. The restore is progressing fine as well. If your VPS or sites on this host machine are not up, please Contact Support and we can look into it for you. – Seohee

Update Mar 21st, 12:50pm PDT: All VPS guest machines are up and resolving without any issues. The restore is progressing fine as well. If your VPS or sites on this host machine are not up, please Contact Support and we can look into it for you. – Oscar

Update Mar 21st, 6:00am PDT: This restore is moving along, though we are finding issues with some guests on this server. We are fixing these one-by-one at this time, so if your sites on a VPS from this host are not up, please Contact Support so we can check it out! -Patrick

Update Mar 21st, 2:00am PDT: The VPS guests have been built and are mounted from backups and the restore has begun. Unfortunately, the server is having some network issues currently and our Admins are investigating. We will update this post shortly with an update to this situation. Thanks for your patience! Please contact support with any questions! -Patrick

Update Mar 20th, 9:17pm PDT: The server is currently rebuilding each VPS guest. Each VPS should come back online using  files hosted over the network from the backup server. Once that has completed then we will begin copying files from the backup server to the local file system. We’ll continue to make regular updates here with the progress. We appreciate your patience. -Dan J

Update Mar 20th, 8:36pm PDT: We are still working on getting the new hardware setup for the move,  and hope it will be ready sometime within the 15 – 20 minutes. We’ll make sure to provide you with updates as we get them. We appreciate your patience. -Seohee

This entry was posted on Tuesday, March 20th, 2012 at 8:01 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

125 Responses to “Web VPS host server ‘homie-vserver194′ moving to new hardware and restoring from backups (RESOLVED)”

VPS guests on homie-vserver194: ps30164 ps33597 ps34881 ps34883 ps34885 ps34887 ps34888 ps34891 ps34893 ps34894 ps34896 ps34897 ps34963 ps35139 ps35144 ps35147 ps35148 ps35149 ps37349 ps37357 ps37657 ps38452 ps38988 ps39353 ps39731 ps40128 ps40988 ps41351 ps41483 ps43740 ps44044 ps44045 ps44329 ps44454 ps47546 ps48000

Thanks for the update. We’ve been down since around 6:41 PST. Any further updates?

@Richard: each VPS guest is being added to the new hardware, it takes some time for them all to setup, but after that your server should be online using backups

Ok, Thanks Dan.

I am in Hong Kong. No access to any mail throughout our work day today. Very frustrating / difficult. When will the problem be resolved?

@JZ: Your site is hosted on the shared web server ‘silversurfer’ and not the VPS host listed in this post. Your site is also up from my location, so please contact support for assistance.

I’m from Brazil, and I’m also very disappointed with this. More then 12 hours and the problem persist. When are you going to solve the problem?

Come on guys. Still down here.

It’s been almost 12 hours without all my sites. This is becoming really anoing.

@wagnoff: Just replied to your ticket and your sites are up!

@Allan Brito: Just replied to your ticket and your sites are now up!

@Jurrien Strobel: Your VPS is not hosted on this host server and is experiencing an entirely different issue. I am resolving that now for you, but please make sure to submit a support ticket for issues like this, as you have none open at this time.

Our VPS ps35144 remains DOA… we’ve got two tickets in for it now, which really puzzles me. It’s just one server, surely you can *test* to determine if you’ve properly restored the sites and not rely on your customers to do your QA testing for you. This Dreamhost “outage of the week” club is getting really tedious.

Our VPS ps89362 is down since yesterday…

VPS ps89362 DOWN

We’ve been down (all clients websites) for the last 12 hours since we decided to upgrade for the VPS service. Dreamhost informed us that we need to wait until the process was completely done. We’re under great stress due to the fact that this is the very firs time we realized this operation and have no idea about how long does it take to be finished.

Once we choose to upgrade our plan we didn’t know this could happen. There was no alert about this situation we get into.

We don’t even know if there’s a chance of undo the operation and get back to our original plan once we started this process.

We really need you guys to come up with a “B” plan, ASAP!

What is going on? ps35149 is still down…

Still down, would like an idea as to when we will be up

When will the categories be able to be loaded? I have lost two solid days of business now. WHEN???

@Mike Renzi – Your site and VPS are up now.

@Opus – Your VPS is not hosted on this host server. Your site is down because you didn’t update your DNS A record for your domain at cloudflare. Check the panel DNS link for your domains A record and once you update it your site will be up again. The rest of your sites hosted here are up.

@Brenda – Your VPS and sites are up right now when I checked them.

@Susan – What categories are you referring to? Your site appears to be working fine when I checked it. Write into support for more help with this.

Our VPS ps35144 remains non-functional. We now have three tickets open with no resolution in sight. I can open a few more tickets if you think that will help.

Our site is down VPS PS79914 for the past hour. I have two open tickets already. Please advise!

No one solution in here, no one ticket respond.. Any suport..

We were up but now we are down again

ps25996 is still down. Two tickets raised, no response!

@Michael: your VPS appears to be hitting it’s memory limit

@Danny: your VPS is online now, though it’s not on this machine

@wagnoff: your VPS is online now, though it’s not on this machine

@Brenda: your site is not on a VPS, but it seems to be working now

@Stewart: your VPS is on homie-vserver135, it’s running a bit slow which was causing some timeouts, we changed some apache settings to hopefully help with that

I wonder why I just went down. Again.

@BT: Your site is hosted on the shared web server ‘ashgabat’ so any issues you are having are unrelated to this post. Please be sure to contact support for that, but I show that your site is online now!

Berlin- site down…

@anonymical: Your site is loading and I see no problems with your hosting. The apache hosting your site even has a 162 day 7 hour and 45 minute uptime. If you wish us to investigate what we can find, please write to support with what errors you are seeing.

This was sent to me yesterday.
NOTHING IS WORKING – again terrible reliability with Dreamhost –

Hello,
We’re writing to let you know that the server that your sites are hosted on, ‘sugeknight’, is having some hardware issues that will result in site downtime. After being investigated by our admin team, we determined that the hardware was un-salvageable and we will be moving this server over to new hardware. We expect to have everything back up and running between 4:45-5:15 (PDT). At that time, the new server should be up and serving files from the backup server so that sites will remain up while we copy the data locally to the server in the background. Our admins will continue to work on this server until they’re sure that all services are back up and running. If you continue to have problems after an hour from now, please contact support and we’ll be happy to assist you further.

Thanks!
The Happy DreamHost Server Fixing Team

umm… sites are down…

My PS is down, ps46383 is down and my main website is down as well for over 1 hour. Please look into it.

@Seay: If your sites are down you should open a ticket through the Dreamhost panel

@Felipe Veiga: I’m checking your VPS now

Here we go again,

All my sites are down right now. Since Wednesday morning when my sites apparently come back from this restore, things got really weird. My uptime record was 12 hours. Every 1 or 2 hours my server becomes unresponsive and apparently it restarts by itself.

I have a ticket open with DH support since yesterday (21 hours) with no response.

Please guys, help me out here!

Microscopium has been down for hours and no one is even acknowledging my support request.

What’s going on?

Everything is dow, the panel as well !

I’m down. POP and webmail. website okay.

I can get to my site, but webmail is down- getting a 500 error.

Geez, POP, SMTP and Webmail down again. Get it together.

All my sites and mail still down, no access to dreamhost.com or any official way to contact support.

My sites on nuggets are down. I was not notified of any maintenance or potential downtime and I am not happy about it. I’m going to start looking for a new host.

Have had other sites hosted on pair.com and have had no issues. Will begin migrating my dreamhost accounts to this host.

Increasingly frustrated by the slooooowness and the lack of access to my site all week.

https://panel.dreamhost.com is down

Kind of scared – maybe I shouldn’t have come here?

Jan, hacking attack issues.
Feb, servers down, network outage.
Mar, HW problem, 15 without mail.

to be continued…?

Websites Still Down ! We have clients getting extrememly upset – losing business…..and no answers on my support tickets…. and yet they claim everything is fine ?!? – can someone respond ?

Still down… this is ridiculous…

March 24th, 2012 at 3:06 pmJonathan Tonge Says:

What is going on? I have no files on the server and I have been taken to a jabber directory when signed in? This just happened in the last hour and it has occurred on all my sites.

Signed in with WebFTP, same thing. Just killed all FTP connections and reset password. Nearly lost my mind – no files.

Then just signed in. Back online. Looks like this is totally on your end. This was painful.

our sites are still down for the last 5 hours, pls look into it. we cant ssh as well, says ‘access denied’. no response to ticket as well.

Is this affecting databases on trad.bennett.dreamhost.com? All of my databases on there seem to be non-responsive at the moment.

our sites are still down for > 13 hours and tickets are not being responded to

Unable to connect to MySQL server

@Bootstrike: I fixed the issues with your VPS, it is back up now. I responded to you.

@kritterbox: I found the issue with your mysql service. I responded to you.

@jfreg: No, this would not affect the mysql server.

Thanks a million! Up and running now :)

yep, up and running finally, thanks.

down twice this month what is going on tickets are being answered please help

25-50 virtual hosts on a machine? What kind of machine is it?

Hello, I am having issues with Joomla 2.5.2 on Qualtech.us & websterbaptist.org. When ever I try to upload files VIA PHP or try to do some other things with PHP I get an internal Server error. I first saw this happening over the weekend. Could you look into this? Thanks

Two of my sites have been down since yesterday; opened up a ticket, but no response. Both are hosted on a Zaniah machine. My clients are getting anxious as they are executing Google AdWords campaigns and driving traffic to a site that won’t even load!

PLEASE HELP!

all my sites hosted on ps15609 are down since friday. i’ve also opened 2 tickets and got no reply. what’s going on?

GOOD JOB DREAM HOST – I JUST LOST MY CLIENT.

STAY TUNED FOR A REQUEST TO CANCEL / DELETE MY ACCOUNT.

WAY TO GO.

Perfect dreamhost, friday you had a major downtime and now you can´t even answer our e-mails.
I really don´t like to tell my clients that I don´t even know what is going on.

Site is still down after 2 weeks. What is going on?

webmail is extremely slow / inaccessible

Damn you and your outages! I pay for these services! I demand a $0.25 refund!

It looks like I am still RO. “* has failed to upload due to an error
Failed to write file to disk.”

Announcement: One of the two hamsters whose wheels power the webmail backend died of a heart attack this morning, resulting in performance problems. The techs are on their way to the hamster ranch to acquire a replacement.

Our email is still down. Please call our main office number. It’s affecting our day-to-day business operations.

Still down… can’t report it using the panel for some reason…. Listen for crickets. I hope your datacenter caught fire and your techs are still inside.

Dreamhost has me chasing my tail here. I need to authorize my login credentials to get to panel to put in support ticket to tell them that my email is down. Cannot authorize without email which is down. AAARRRRRRRRRRRGGGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHH

Congrats DreamHost support! I´ve found out what was going on before you… And believe me, that was not fast.

I don´t know why, but commenting my “Options +FollowSymlinks” fixed the problems I had, and mod_rewrite is still working just fine.

If someone at Dreamhost decides to change a major config like this, the minimum they should do is let they users know about it, with some time to make the required changes!

Our site is down again, 2nd time in as many days. This is not cool.

Anyone here with there site up but with it giving 500 & Internal Server Errors?

@vaughan I was. Check it out how I’ve solved it above.

When will aquaman work? Its incredible SLOW!!!! PLEASE HELP!!! I have some work to do!!!

And just to help anyone else having problems with 500 internal server due to Dreamhost changing the configuration without warning its users, here is the command that I used to get things up again (with a admin user):

sudo find .* -name “.htaccess” -exec sed -i ‘s/Options +FollowSymLinks/#Options +FollowSymlinks/g’ {} \;

and

sudo find .* -name “.htaccess” -exec sed -i ‘s/Options +FollowSymlinks/#Options +FollowSymlinks/g’ {} \;

Enjoy it.

Can someone explain what is going on? Emails were taking 3-4 hours to come through yesterday and I just found out from a client that emails are now bouncing back today saying the server is down and cannot deliver to us. This needs to be fixed we’re falling behind and it looks like were ignoring emails when in fact we’re not receiving them.

Woolman.org is down. DNS won’t resolve. Did you just migrate it to the new server and forget to update the DNS?

I’m broke, guys. Help ! :(

All my webs hosted in DREAMHOST are HACKED!!!

Whats happen??? I am desesperated!!!

@Coleman, @Richard: I’m working on your VPS’s right now and will get back to you via email shortly.

My site has been down for 5 days. Help please!!!

totally unrelated-but its been almost 24 hours since issue reported and NO response from support. The same issue on another account, reported more than 8 hours ago-NO response.

You are going downhill, and fast dreamhost.

24 hours – down – no response- 3 tickets submitted (i know, what good is 3). Can I at least get an update?

Richard

Our site is down again, after being up and down repeatedly, and unpredictably, since this move first started.

We have re-booted numerous times…no luck.

We’ve submitted support tickets…no luck, and no response!

Can someone at DH please check our site???

our site is still down….
uptime ratio sucks..

lost big amount of money in last 2 months…………

cant get into my email

ya.. cannot access email here now either.. +_+

Al my mail is down. It has been spotty for two days and is now out for the count. People received undeliverable reports back.

You all are pathetic. Stop whining and speak with your feet. We moved our site and all of our problems are gone now. If you have been complaining for months about your site being slow and DH just told you everything was fine. Well, we know now it wasn’t fine. Our speed issues are gone.

http://www.inmotionhosting.com

I am writing for the first time in 9 years hosting websites with DH. What is happening last month or so is just way too much. Website is down for hours each week. What the fuck people? When customers like myself are getting seriously annoyed it means that something is wrong with your business. Get it fixed before your loyal customers move their websites somewhere else.

Getting “ERROR: Connection dropped by IMAP server” on webmail

POP / SMTP just woke-up.. seems (knock on wood) Ok here again FTM…

Ticket #4891253 Website down, jabber down cannot ssh in. Live chat support offline.

March 28th, 2012 at 9:47 pmNicole Neditch Says:

I am still down. No updates for hours. What is going on??

My sites don’t work since 3 days.. on sainthelier server.. whats the hell is goin on!!!!!!!!!!!

My sites are not up — Your updates say that they should be — I have sent two support tickets and still NO response. Please let me know what is going on.

3 days off, and my VPS has only one website restored, all the others are down.

Where are the others files from my other sites?

Just seeing if this status board is being checked as it appears that support requests are not. At this point my mind is made up and I’m moving as much as my business as I can away from Dreamhost, but I have to say that a big reason for this move (aside from the catastrophic issues we’ve had this year) is the way you’ve handled it customer service wise. I’ve been with you guys since 2005 and you were one of the few things I never had to worry about web development-wise. At this point you guys appear to be over your head and worst of all you just don’t seem to care how it effects your customers. Chat no longer works, I see in these comments people who have sites have been down for days. On top of that this status is over Homie 194 and I’m having issues with Homie 164 — and I’m not sure if this is typo or failures on multiple servers.

“Just seeing if this status board is being checked as it appears that support requests are not. At this point my mind is made up and I’m moving as much as my business as I can away from Dreamhost, but I have to say that a big reason for this move (aside from the catastrophic issues we’ve had this year) is the way you’ve handled it customer service wise. I’ve been with you guys since 2005 and you were one of the few things I never had to worry about web development-wise. At this point you guys appear to be over your head and worst of all you just don’t seem to care how it effects your customers. Chat no longer works, I see in these comments people who have sites have been down for days. On top of that this status is over Homie 194 and I’m having issues with Homie 164 — and I’m not sure if this is typo or failures on multiple servers.”

here here, I’ve already started to move my clients to my own reliable reseller hosting (see my name for link). I used to rely on you guys for a strong hosting platform and great support, now I’m lucky if my tickets even get answered within 1 day or more. Shame as you guys used to be great :(

I’ll also move my sites back to hostgator, a live support of a hosting provides is inevitable for us…

I’m about to pack my things as well. I run my business off of DH and with my site down for over 2 days and no responses I have generated 0 ZERO income, unless DH wants to pay my bills, I’m out of here.

Still down. three support tickets logged. No response from dreamhost. This is ridiculous!!!!!

Unacceptable, my VPS ps24467 down since yesterday plus no support at all. 1 day downtime and counting, wtf?

Why do all of your updates say that everything is working when it is not? I would much rather you guys be straight about the problems and when you think they will be resolved. I am sure that I am not the only one who has clients calling. Without updates from you, I have nothing to tell them! And my tickets remain unanswered!

Right there with you. Our VPS machines have been down since yesterday since homie-vserver164 crashed. They don’t even mention it here on the status page. There is a note on the panel, but it’s basically useless since as of the update four hours ago they claim that all VPSs should be back online… of course, ours still aren’t. And I’ve yet to receive a response to our clearly useless support ticket.

It’s clear that DH no longer has the required support staff to handle all their problems, they’re just overwhelmed. That can be excused for a while (i.e. sudden spikes in outages, etc.) but this has been going on for months now. There is no valid excuse at this point.

My website is down for more than 24 hours! I am not expecting this to be fixed anytime soon since all the support tickets are simply ignored.

Hope my feedback will help someone to make right decision. DO NOT SIGN WITH DREAMHOST!

Dreamhost people, your loyal customer (8 years plus) is moving his site to new hosting company. Simply sick of it. Well done DH! Keep losing people who made you successful.

Amen JD.

I always advised clients to go with Dreamhost, as they hit a good point between features and price. I’m now regretting that as the last 4 months have been SHIT.

My backups user hasn’t been accessible for TWO MONTHS, then our VPS fails on top of it. Usually I’d say “no sweat, I’m diligent and have backups.” Well in this case, my automated backups are failing.

Good job Dreamhost, we have 40 websites hosted here and thats soon to be as few as we can manage. I won’t *ever* recommend this host to another client.

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Asi desde hace 24hs, y no hice ningun cambio en mi sitio… que esta pasando ??

I can’t access my WordPress Admin. Please Help!

My sites on my VPS have been down for over 24 hours. To be polite, my clients are “very upset”. I’m very upset. I have been a loyal client of DH for over 7 years, but I am now looking for other hosting.

My sites are down… again.

So after a full day of having a VPS down, I finally got an email said “Things are back up and running now, and I’m able to create files on the file system”. Needless to say, I don’t have FTP access and my sites still aren’t up. Then I got a message that my tickets were being put in a “Severe Problem Queue”. Even though I’ve totally given up on DH, I continue to be disappointed by how they’re handling this.

I personally find it funny that this status site carries the disclaimer: “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

Funny, I thought that was the same policy as their support email.

Funnier is that I can’t move to another hosting company because my files are trapped in Dream Host.

@George: It’s ironic… the last few times I’ve had problems with DH, the only way I got a response is to post a complaint here. Support tickets got zero response until I posted a gripe here. I used to get great response with their chat feature, but they shut that down months ago.

In line with this pattern, I got an email last night after my post above saying that my ticket was getting assigned… unfortunately, that was last night and as of this morning nothing has changed… VPSs are still down, and no updates from support. Panel note still says that everything should be online.

Sad thing is I don’t want to post a gripe here… but DH is forcing me to by apparently ignoring their ticketing system and disabling their chat support. At this point I find it hard to blame the support folks… my sense is that they’re simply overwhelmed with too many things to fix and thus are simply fixing the squeaky wheel. I blame the management that’s not applying the necessary $/resources to hire more support staff and replace bad hardware. And every one of those cheery monthly newsletters from the CEO are only slaps in the face to a decade plus customer who’s tried to remain patient but rapidly approaching his limit.

Again, all of this is ridiculous… as I noted in my response to their email last night… I could have built a server from scratch at this point, not to mention simply restoring a VM image. Insane.

This is it. 3 days with no sites active. I am so off Dreamhost. I should have left after the last outage. June would mark 10 years with DH … but I am through with this bullshit.

@JD

Certainly true. My issue is not that they had a hardware failure. Shit happens.

My issue is that we don’t even have our own status posting for the host machine and effected ps. Its like they are trying to hide some of the outages because this blog is stuffed full of catastrophic failures and it looks bad.

My issue is that the information noted in the panel is inaccurate (as you noted.)

My issue is that *even the panel* has had significant downtime recently. In fact, the panel is down for me *right now* as I type this – no joke. Well, its a joke, but the joke is on Dreamhost.

My issue is that my backups user hasn’t worked for 2 months. They could have simply assigned me a different user – they KNEW it was congested and essentially down. Hell the panel reflects 0GB usage right now! If my backups server was working, I wouldn’t be having an issue right now. Multiple tickets were just brushed off and closed without an actual resolution. No doubt it looks bad to have tickets for broken shit stay open.

This was the response on the backup server issue: “Sorry for the trouble. Hanjin has been really sketchy lately.” NO SHIT ITS SKETCHY! YOU DON’T SAY.

The follow-up: “Unfortunately for the time being, the backup server will be slow until more of the moves can be run and there is no ETA for when that will be complete.”

My issue is that we got an email with the following from Dreamhost support:

Things are back up and running now, and I’m able to create files on the
file system: psXXXX:/home/user# touch blah.txt

Its like holy shit seriously? Did you try to visit ANY SITE on the VPS? I can’t log in via SSH with any user, and none of the sites we have are up. Thats troubleshooting 101 – “test resolution”. Go ahead, pop open the apache vhosts and grab a random site and see what happens.

Several we’ve run into some obscure problem that prevented us from accessing mysql (which of course kills all of our CMS based sites). Usually this is met with “Are you sure its not a problem with your application?” Yes, all 20 of them stopped working simultaneously! I just respond: “Then explain why >mysql -u user -p -h mysql.hostname.com doesnt work?”

Every month for *months* I get downtime notices from Pingdom, and I only monitor the VPS with our “important sites”, not the one that died here.

2012-08-01 dh VPS 99.92% 0h 30m 00s
2012-09-01 dh VPS 99.97% 0h 15m 00s
2012-10-01 dh VPS 99.93% 0h 30m 00s
2012-11-01 dh VPS 99.90% 0h 45m 01s
2012-12-01 dh VPS 99.62% 2h 44m 58s
2012-01-01 dh VPS 99.83% 1h 15m 00s
2012-02-01 dh VPS 99.90% 0h 45m 00s

Is an hour long outage a big deal? Not really, but when it seems like you’re explaining away outages to clients *EVERY MONTH*, they get really touchy. Drop a multiple day outage on top of it, with the misinformation and MIA backups user and you’ve got a toxic issue. Let me tell you, when you have 20 low traffic websites on a VPS, *someone* is going to notice that outage.

At this point, I just cant afford to defend Dreamhost further, as its damaging my reputation with my clients.

Two days having VPS down on homie-vserver225! Two days! Support does not exist. Support tickets are sitting there for days. Posting here is the only hope I have to get someone’s attention.

If you can move your site somewhere else today (having current copy of your database) just do it. Do not wait until it’s too late.

We’re planning on leaving Dreamhost soon.

Our vps on homie-vserver225 is also down and has been for over 24 hours. I need help a.s.a.p.

If the VPS cannot be restored to working condition I need our domain redirected (or cloaked) or for our site to work off of a shared directory. Our domains on the shared servers seem to be working fine. Even if we could get the domain to work from a shared server that would be ok. I have not been able to get our domain to respond to any changes: redirect, switch to shared server account, etc.

If any Dreamhost staff reads this please help us out.
Thanks.

At this point reading all of everyone’s comments — and hearing nothing from Dreamhost — I’m basically working from the standpoint that my websites are toast. Out of desperation, I tweeted @dreamhost and @dreamhostsimon to plead our case but I’m assuming that will also fall on deaf ears.

Well here’s some positive news, my tweets got a response from @DreamHostCare in less than an hour!

“Please believe me, it’s had all our attention since it started. I see what needs fixing with yours and will email you asap!”

My sites are still down, but at least it’s from a new error. As Twitter seems to be a channel that DH is responding too, I encourage you to do the same.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Network issues in Irvine datacenter causing delays to server updates (RESOLVED)
4:51 pm

Network issues in Irvine datacenter causing delays to server updates (RESOLVED)

Posted (March 19th, 2012 at 4:51 pm PST) by Dan J

Our Irvine datacenter is currently having packet loss issues with our backend network. This is preventing some updates from the Dreamhost panel to servers in the Irvine datacenter. This includes things like FTP user creation, password changes, changes to website configurations done through the Dreamhost panel.  This is also causing some problems with dedicated servers ability to reboot properly so we recommend that you do not reboot your dedicated servers at this time. The existing websites should still be online as the front end network is working properly. Our admin team is working as fast as they can to bring the backend network online again. All updates made in the Dreamhost panel should all go out once the backend network access has been restored. We apologize for the delays.

Update Mar 19th, 6:05pm PDT: The issue has been resolved now. If your site appears to be having issues still, please contact support. – Dan J

This entry was posted on Monday, March 19th, 2012 at 4:51 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Network issues in Irvine datacenter causing delays to server updates (RESOLVED)”

All our websites and client websites are down and have been down for over 12 hours. Can someone please fix it.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver135′ moving to new hardware and restoring from backups (RESOLVED)
5:34 am

Web VPS host server ‘homie-vserver135′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 19th, 2012 at 5:34 am PST) by chalise

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver135′, we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver135′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 24th, 4:05pm PDT: This restore process is now complete. If you are experiencing any issues with your VPS, please Contact Support for assistance! – Oscar

Update Mar 24th, 5:05am PDT: This restore process is almost done. Our admin team is still doing some final manual restorations. All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! – Dan J

Update Mar 23rd, 8:55pm PDT: This restore process is almost done. Our admin team is still doing some final manual restorations. All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! – Dan J

Update Mar 23rd, 9:50am PDT: This restore process is almost done. There are a few guests that are manually being restored by our system administration team. All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! – Oscar

Update Mar 23rd, 12:50am PDT: This restore process continues to execute without issue. All VPS guests on this host are reporting online. If you are experiencing any issues with your VPS, please Contact Support for assistance! -JJ G

Update Mar 22st, 10:30am PDT: The restore continues well and all VPS guests are up and running fine. The load has dropped since previous and services should be faster. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Justin

Update Mar 22nd, 4:10am PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Patrick

Update Mar 21st, 10:30pm PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Seohee

Update Mar 21st, 12:50pm PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Oscar

Update Mar 21st, 6:16am PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Patrick

Update Mar 21st, 2:07am PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Patrick

Update Mar 20th, 6:24pm PDT: The restore continues well and all VPS guests are up and running fine. However, the load is higher than we would like on the new hardware due to the restores and VPS running off the backups, so we are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Seohee

Update Mar 20th, 11:00am PDT: The restore continues. The load is higher than we would like on the new hardware due to the restores and VPS running off the backups, and are seeing what can be done to improve the situation. The speeds may be slower until the restore gets further along, unfortunately. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Justin

Update Mar 20th, 5:10am PDT: The move to new hardware is complete, all VPS guests are running off of the backups, and the restore from backups is continuing to run smoothly. If you notice outdated or missing data on any of your sites, or are experiencing any other issues, please contact support so we can look into it for you. – Chalise

Update Mar 19th, 8:00pm PDT: The move to new hardware is complete, and all VPS are running off of the backups. The restore from backups seems to be going fine. If your VPS appears to be having issues still, please contact support. -Dan J

Update Mar 19th, 11:30am PDT: The move to new hardware is complete, and all VPS are running off of the backups. The restore from backups to the webserver has begun. If your VPS appears to be having issues still, please contact support. – Justin

Update Mar 19th, 9:30am PDT: The move to new hardware is complete, and all VPS but a few are currently running off of the backups. Our admins are working on fixing up the last few VPS that are having issues after the hardware replacement. The restore from backups to the webserver has begun. If your VPS appears to be having issues still, please contact support. – Justin

This entry was posted on Monday, March 19th, 2012 at 5:34 am and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

97 Responses to “Web VPS host server ‘homie-vserver135′ moving to new hardware and restoring from backups (RESOLVED)”

Affected VPS Guests:
ps21397 ps21876 ps21895 ps21930 ps21958 ps22005 ps25381 ps25482 ps25593 ps25638 ps25819 ps25996 ps26003 ps26033 ps26093 ps26109 ps26262 ps26275 ps26355 ps26421 ps26458 ps26631 ps26753 ps27449 ps27791 ps27979 ps28737 ps29489 ps29532 ps29577 ps29910 ps34309 ps35370 ps35768 ps36288 ps36667 ps37527 ps37845 ps38141 ps41463 ps42489 ps43585 ps44284 ps44285 ps48969 ps49285 ps49363 ps54610 ps55855 ps55865 ps56823 ps73156 ps74123 ps80740 ps82845 ps82970 ps83205 ps83760 ps85332 ps87106 ps89495 ps89719 ps90118

3.5 hours in and still down :-(

Please give more details. We need to know how we are getting on and how close we are to a resolution.

Echoing David’s comment, you surely have more details. How far through is fsck? When does it estimate it will finish?

Ahh, redundant servers saves the day.

If using Dreamhost always make sure that you have a redundant server. Now if only I would listen to my own advice and actually implement full mirrors.

How much longer until fsck is done? Is there really no progress indication at all?

@David & @Marko & @Jim: Sorry for the delay, the file system check would take too long to be acceptable downtime, we had to do a hardware replacement. The new hardware just got done building all the VPS guests and mounting the backups. We are fixing up a few stragglers but most VPS should be back up and running now. If you still see issues, let me know the VPS name and I can see what I can find.

it’s more than a little frustrating (understatement) to continue to have these interruptions. I’m perplexed -why so many since Jan?

Hi Justin,
I am still waiting for ps35370.
Any news?
Thanks,
David.

Still waiting on ps89719 as well.

@emily: This is an issue with a single VPS host machine, it does not have anything to do with any of our 1000’s of other servers. homie-vserver135 has not had a move to new hardware previously. If you are having a problem with your VPS, I can take a look if provided the VPS name.

@David: It should be up and running now.

@Ian Neubert: Checking out your VPS.

Thanks Justin.

I’m unable to log in to any of my shell accounts on ps37527. It’s responding to pings and I’m getting an ssh prompt but unable to authenticate.

Hi Justin,
I am still waiting for ps55855
Any news?
Thanks,Giorgio

Hi Justin –
Still waiting for ps26109. Sent in a ticket over an hour ago and no response yet.

Hi Justin,

ps27791 can be pinged but neither can i get in through ssh (invalid pw) nor can I bring anything up via http… Please let me know where we are with the restore. Thanks!

My redirected email is still not working. My direct email is OK – I just tested it again. This problem has persisted about 20 hours now. My support ticket is:
Mar 18, 2012 – 23:53:02 / #4825137

Any explanation or expected time for a fix?

Oh, and another thing – it seems that my server logs have not been updated since 3/16/2012.

And that seems to be a nearly blank file. The backups seem to stop as of 2011-08.

OMG! LoL!
DH is dying slowly! I am going to Europe!

Do you have Pentium III on LAX?

Still down, what’s going on?

@Jim I test a few users and did not encounter any login problems. If the problems persists, please submit a support request and include the details.

@Aron I had to restart your VPS. All services have been restored.

@scidox Your VPS guest in online.

@Marko Your user directories are accessible. If the problems persists, please submit a support request and include the details.

@Tom Could you please provide us with your VPS guest?

@Tom Says:
Still down, what’s going on?

They are adding more RAM on shared hosting! (From 90 MB to 120 MB) LOL
Thank you Dreamhost for your RAM limits !! You are great, I will try to find a good hosting company asap!
All my websites have more than 1.5 seconds! This is awesome. LoL

Keep up the good work!

My redirected email is STILL not working after almost 24 hours. I did the test as suggested and it reported no errors. My support ticket has not been answered, and I have found no announcement of any progress or even indication of this problem. Everything else seems to work, although the logs seem to have disappeared and are not being updated.

My account renews in April. I think I’ll be looking for another provider unless I get some feedback and resolution soon. An entire day without incoming email is costing me a lot because customers cannot reach me and I’m spending too much time trying to get information and a resolution to this serious problem.

Apparently things went down in January but they were restored quickly enough. But this is inexcusable. There is no phone support unless I pay for it. If Dreamhost is willing to pay for my downtime and lost revenue then I’ll gladly do that, but I doubt it will get any better results.

Hi Justin,
My VPS – ps35370 – came back up but has disappeared after a couple of hours. I don’t know if this has anything to do with the previous issue but we are offline.
Please help
Thanks,
David.

@David Your VPS guest is back online.

OK, my emails finally arrived. Thanks.

What’s going on!!!

I don’t think it’s related but my site has been up and down all afternoon. Now it’s down… again.

I’ve submitted a ticket more than one hour ago but no reply so far…

Is it a general outage or a problem with my site only?

Help!

My site has been down for hours and no status info on it? Why has my service from you gone down so much? Really disappointed!

Justin,

ps27791 is down again… Please take a look at it. Thanks.

@brians – I’m still unable to authenticate via ssh on ps37527. I have submitted 2 support tickets which are as yet unanswered. i’ve tried changing passwords via the panel and rebooted the server without success.

SITE is up finally. Thank you! Not sure why you did not know that the site was down to begin with.

PS6215 — back not restored properly. can you look into it ?

thanks

Now up to 3 unanswered support tickets! I can now log in to ps37527 but it has been restored from an out-of-date backup.
Please respond either to my support tickets or this post as soon as possible.

ps28737 – still down, can you please look into this

@Marko: your vps appears to be online now, I increased the default max_connections limit in Apache because it seemed you may have been hitting that limit

@Jim: one of our techs just replied to your tickets

@Dave: ps6215 is on homie-vserver20 not 135, but it sounds like we’ll need to restore your files manually from the old disk, I’ll have someone reply to your tickets

@Ben: ps28737 should be online now, though it’s mounting from the backups still and those backups might be missing some of the data, we’ll likely need to restore some of your users from the old disks

@ Dan J, All my sites on this server are still down after around 18hours. Can someone please sort it out.

@Ben: I’m in the same boat as you. Support said they will restore my users from homie-vserver135 but they didn’t say when or how long it might take.

@Dan J – any clues? I requested a callback in my last ticket #4827487 but haven’t heard from you yet.

Hi,
My VPS – ps35370 – is down again, and has gone down a few times in the last 12 hours.
I don’t know if this has anything to do with the previous issue but we are offline and can’t carry on like this.
Please help
Thanks,
David.

@ Jim. I think you will agree this is horrid. We are now pushing 24 hours downtime soon which in the world of hosting is pretty sh#!

@ Dave. The communication from your support is sub par. Please let us know what is going on. How about a live chat or some kind of support other then automated emails. Surely you can manage this?

New machine, more RAM added or external bandwith? ;)

Hi,
Looking at the graphs of My VPS – ps35370 – it is clear that everything went wild since the change. The toll on cpu and memory resources is very clear.
Please help
Thanks,
David.

@Ben: yep, a further 2 unanswered support tickets and a callback request with no call so far.

@Dreamhost Support: Please respond and let me know what you are doing so that I know whether I need to restore my web applications myself!

OK this is just incredible… email message from “The Dreamhost Ticket Moving Robot”

“This is just a notification that your message has been moved to our severe problem queue. This queue is for major outages that cause a large amount of support to come in. Our goal is to resolve the problem based on the information you have provided. We will then send you an update once the issue is resolved. Depending on the problem it may take longer than 24 hours for a response. Please hang in there, when messages are moved here your problem is given top priority. Please note that callback support is not available for messages moved here.”

This message reads like a blow-off. How on earth does giving my problem “top priority” equate to me only being able to expect an update “once the issue is resolved” with “callback support not available” and possibly having to wait 24 hours or more. I completely understand the pros/cons of hosting at Dreamhost’s price point but when an issue is recognised as “severe” I need more communication, not less! We’re at 26hours downtime and counting, which for our small shop is critical at this time of year. I could restore the site from my own backups but it’s no good me doing that if Dreamhost are doing the same thing! I need to know is what you are planning and when you are planning to do it and I have absolutely no way of contacting you to find out.

Does anyone else think this is a sane communications policy?

@Jim: I’m working on your issues right now. Your larger amount of data is preventing us from easily copying your missing data from the old hardware and also means your VPS will be one of the last ones to be restored from backups. I’m looking into some alternative methods to get your missing data restored and your VPS back in business. Please standby. I’ll send this message via one of your tickets as well so you know what is going on.

Justin,

ps27791 is down again… it’s actually been going up and down and up and down… But now it’s been down for more than an hour. Rebooting doesn’t help. Please look into it!

@Marko: Ok, seeing what I can find.

Hi Justin,
My VPS – ps35370 is experiencing what Marko has described. Rebooting doesn’t help. Please look into it.
Thanks,
David.

@Justin

Thanks!

No connection via http, nor via FTP. Can’t connect to the MySQL either.
The site is totally down for 90 minutes and has been offline on and off for over 24 hours. Reboots and reallocation of resources don’t make a difference.
please help. Please communicate.

Hi Justin,
VPS – ps55855 is experiencing what Marko has described. Rebooting doesn’t help. Please look into it.
Thanks,Giorgio

Justin, please reply. 2 hours and 40 minutes of complete downtime (on top of the on and off downtime). Not knowing what’s happening is the worst thing. I would greatly appreciate a clue as to what is going on and how much longer the site will be down.
ps35370
Thanks,
David.

Hi Justin –
I was given an option to create a new vps and have everything moved over (from ps26109) but it looks my response to the support ticket was misunderstood. It seems like the support tickets are taking a long time to get a response back and I would like to get things moved over as soon as they can to avoid further downtime. Can you help with getting this communicated?

Justin, did I understand Aron correctly? Would it be possible to move everything to the new server and resolve this?

@Marko ps27791 is now online,and we’re working on getting your sites back up and running.

@David We’re looking into the issue with ps35370 and will be in touch shortly via support ticket.

@giorgio ps55855 is now online, and we’re working on getting your sites back up and running.

@Aron We’ve picked up your support ticket and you will receive a reply shortly.

@Marko, All we can do at this time is continue the restoration to the new hardware.

Hi Brandon,
When you say you will be in touch shortly via support ticket, I sure hope you didn’t refer to what I just received via support ticket, telling me that “a notification that your message has been moved to our severe problem queue… Depending on the problem it may take longer than 24 hours for a response”.
I’m dying here.

@David Nope, I meant from a real human being. :) You’ll hear from us shortly, we’re still looking into the issue.

Thanks Brandon,

I’m going to submit a support ticket about this (the last few tickets were from yesterday) so that I can keep closer tabs on what’s going on. Thanks!

Brandon –

Thanks for the clarification. Anything to get my site functioning again would be appreciated. Like David said, im dying here!

Is there anything on the Dreamhost wiki about what I could have done differently? Backups I should have had? based on the previous issues of this nature it looks like this could take a few days(?!) to resolve? I would hate to be caught with my pants down again in the future if something like this comes up.

Thanks Brandon.
Waiting with anticipation.

My site is too slow, take too long to respond!

I have been having all sorts of issues the past couple days, not the least of which is my sites being down repeatedly over the last few hours. There was a huge memory spike for seemingly no reason today, and it’s all just been overloaded. Rebooting takes up to 15 minutes, and has worked off and on, but it keeps happening. The sites seem to be working now, but who knows if that will last.

I’ve also had issues with a domain transfer where created FTP users simply would never take, and just seemingly take forever to actually register without doing so, meaning I have no access to those sites until they do. I finally was able to get one to go through after hearing back from support (which took almost 24 hours, also atypical for Dreamhost), but I am hesitant to try to use anymore.

My VPS is ps88651.

Are all these issues related? I never had issues like this until moving to a VPS, and I can’t really justify the increased cost if it’s going to give me inferior performance and service.

Can this affect the shared hosting?

My seven domains (shared hosting plan), down 3 times TODAY ONLY!

Hi Brandon,
We were online for a while (albeit slow) but gone down again.
I am being hassled for answers and I have none.
Can you tell me anything? Possibility of a move to a different server? The cause of the problems? Any timeframe?
Thanks,
David.

I thing for many of us, our server is critical to our business. I have over 50 domains on my VPS and now have been down for over 30 hours continuously. As a result I am currently up at 4am in my timezone checking to see what is happening. This is definitely killing my businesses. These were critical days for some of the websites. One website has already lost $30k of sales, let alone the damage done to the brand and customer experience. That is only one of the websites.

Dreamhost I think if you could at least give us HOURLY updates along with an updated eta everyone would appreciate it. Currently I feel in the dark. I have been responded to a few times by support and they say they will get back to me but never do.. that was 6 hours ago… have they forgotten, has something gone wrong, what is the update? Do I have to keep posting on this forum to get a response? Please update us all every hour.

My VPS is ps28737

Yay, I have a few of my sites back :) but none of the important ones… :(

@Aron Yes, having a backup stored locally would help to minimize downtime. I strongly suggest that you begin backup your data and store it locally.

@John Arce This is likely due to the data moves being performed. However, to better assist you, could you please provide us with your VPS guest ( ex. ps12345 )

@Jeffrey Your VPS guest is not setup on homie-vserver135. Therefor you are not affected by this outage. Could you please submit a support request and include the details?

@Neemias This outage only impact VPS guest that are setup on homie-vserver135. Therefor you are not affected by this outage. Could you please submit a support request and include the details?

@David Your VPS guest in online. Unfortunately, moving you to another guest is not an option at the moment.

@Ben Your VPS guest is online and mounted to backups. If you are having problem, could you please submit a support ticket and include the domain(s) that aren’t responding. We will gladly take a closer look.

Also, at the stage where the new hardware ix up and successfully mounted to the backup machine, and the restore is running smoothly, we only update every 4-5 hours. If we notice any problems with the restore or extended downtime on the host machine, we will update the status post immediately.

@brians – can you please have a look at support ticket #4829482 for ps37527.

I need a further update from what @Justin K was doing this morning.

Thanks very much.

@ brian, I have submitted a new ticket #4829628. It lists all the websites which are down which are URGENT, if you can look into this and email me. Once these are done, I will test the more non urgent sites.

Thanks in advance

Can someone please respond to my ticket #4829628

@Ben: we’re working on your ticket

@ brian, no, unfortunately ps35370 is not really back. I click a link and nothing happens for 60 seconds (!), I can’t do anything. The site is not on and it must be on (as I am saying for 36 hours now).

@ brian, after yet another reboot I manage to connect via FTP. Via http I manage to connect only to non-php files and I cannot access phpMyAdmin. I think there for that the problem is with the database server (or the php engine). ps35370.

@ brian, Currently everything does work, but extremely slow. 20 to 60 seconds for a page to load. ps35370.

@ brian, Now it’s just 500 errors on ps35370.

@David: Our apologies. We are working with our Admins to control the load on this VPS host right now. Please be sure to get a ticket in for support so that we can be sure we are addressing all of your issues. It is difficult to provide support here in these comments.

@patrickm, Thanks. There are support tickets open but I don’t really get responses on those open tickets. This issue started day before yesterday and must be resolved. If loads aren’t controlled after two and a half days, new measures must be taken for a full resolution.

@patrickm, Hi,
Now, on top of the fact that the site is down, when I try to log in via FTP I am told that my login details are incorrect. I sent a ticket about this. Please let me know as I wanted to upload an index.html apologizing for the downtime.
Thanks,
David.

@David: Just replied to your tickets! Please see my response for details!

Brian – Yes, I have a backup of my website and it is currently on my other VPS. I have been trying to get my website switched over but something does not seem to be taking. I am assuming you are saying this is the best solution then(have your own backup, and put it on a different VPS manually so it does not need to copy all that data while the restore is happening)?

@patrickm, Thanks. I tried replying but your incoming mail server sent me an error so I had to open a new ticket for the reply.
When will the files and db be back?
Thanks,
David.

@patrickm, Patrick, I’ve likewise tried replying through support but got an error from the mail server. So I’ve specifically created a new ticket related to outdated backups.

@patrickm, you wrote an hour ago, “we will be happy to try to recover your data from the old
hardware!”. Please, I am waiting for 48 hours for the site to work properly. Please go ahead and recover the files and database.
ps35370
Thanks.

@patrickm, 3 hours ago I got your message saying that you moved my domain and that soon you’ll recover your data from the old hardware! I am really anxious for this stage to be completed – this site is offline for more than 50 hours now!
ps35370
Thanks.
David.

@Aron – What Brian meant is that you can restore your files to your current VPS when outages like this happen. Since the restore is going for all VPS guest machines it can be a little slow, but if you have a local backup of your sites you can upload them instead of waiting for the restore to get to your server.

@Marko – This has now been fixed.

@David – I restored the content for you.

@Oscar – You didn’t. There are just 22kb worth of files on my domain!

@Oscar – This is just so ridiculous. I was promised 7 or 8 hours ago that the files will be restored. An hour ago I was told that they were restored. Of course nothing is restored! This is dreadful. The site is offline for two and a half days. I am losing business, I am losing credibility. What is going on?

@Oscar – I sent you an update just now, as a reply to a ticket. tracking # is 4831342. Thanks. David.

We raised tickets about this asking for an update and no-one’s responded to either of them. We’ve been patient and waited over 24 hours, so not like we’re pushing. Some files are missing too.

Would be nice if someone could acknowledge tickets as well after a day or so. Just so we know you care (and a rough ETA) #4831209, #4829629

@David: working on your tickets now

@Ashley: I’ll have someone check out your tickets right away

@Dan J – Thank you.

ps25996 is down again

so is ps37527, not responding to ping or panel reboot

@Stewart, @Jim: It looks like there was an outage during the restore process around the time you two wrote your comments. Our admins already have the guest machines responding and I’ve verified both your machines are serving sites. Please let support know if you continue to have trouble.

My site, subversion and everything have been down for more than 24 hours. This is unacceptable. Please get my site back online so I can get my data and do business with someone who’s serious about this shit.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity problems on our backend (RESOLVED)
4:22 pm

Connectivity problems on our backend (RESOLVED)

Posted (March 18th, 2012 at 4:22 pm PST) by danwy

We are currently experiencing what seems to be some networking issues on our backend infrastructure. This is causing mail login issues for a small portion of our customers, as well as causing the webmail interface to become unavailable. We already have a team of admins on the job to diagnose the issue and resolve it. We don’t have many details at the moment, but we will update this post as soon as we have any additional information about this incident.

Update Mar 18th, 5:20pm PDT: We have temporarily resolved this issue by routing network connections around the defective switch. We’re still not entirely sure what caused the switch to fail, but our Admin team will continue to investigate it further. We’re still in the process of replacing this defective hardware, but we are doing it behind the scenes so that it does not affect any customer services. We will fully test the new switch hardware before we route traffic through it again. You shouldn’t notice any further downtime related to this incident. If you experience any further problems with email or webmail, please Contact Support.

Update Mar 18th, 5:00pm PDT: Our admins are still working on this switch, but have been able to get mail and webmail back up and running. We’re still investigating the cause and will post more information when we have it. Please contact support if you’re noticing any issues with accessing email. Site/VPS outages are not related to this issue and you should Contact Support if you are having issues with those services. -dan w

Update Mar 18th, 4:45pm PDT: We’re still waiting to hear back from our admin team if the switch replacement was a success. Unfortunately, we don’t have an ETA on when this will be resolved at this time, but as soon as we know we’ll update this post with that information. If it only requires the switch to be replaced though, we should be up within the hour (hopefully much less). -dan w

Update Mar 18th, 4:30pm PDT:  We’ve narrowed this down to a faulty switch in one of our datacenters. Our admin team is in the process of replacing it now. We hope to have more information on how this is going in the next 15-20 minutes. We’ll update this post as soon as we have any additional information.

This entry was posted on Sunday, March 18th, 2012 at 4:22 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

29 Responses to “Connectivity problems on our backend (RESOLVED)”

Both our webmail login and our webmail setup pages are dead.

It’s been quite the month(sssss) So many problems! Arghhhhhhhhhhhhhhhh.

I’m in the “small portion.” Hooray.

Sigh. Yet again… C’mon guys…

My VPS is down (as well as 4 hours last night)

‘homie-vserver159′ – ps25767

it will be nice if there was an expected time when it will be fixed.

Is it going to be fixed within the hour? in two days? When?

expected completion time ????

It’s amazing how often switches seem to die, considering they have no moving parts they must be the most unreliable devices ever.

Get your act together. After all the outages and security flaws, I have to consider taking my business elsewhere.

No EMail for me.

Part of the small portion here as well. 2012 has been a rough year as a DH customer…

My site is down as well. :(

Dreamhost? More like outagehost.

I thought web hosts had redundant servers and switches, how does one switch malfunction bring everyone and their brother down?

Moonbeam is experiencing timeouts.

Count me in. The last few months we’ve had these problems. Will accounts be compensated?

yeah im in the small portion as well, does seem like DH has been having alot of problems recently, might have to start shopping around

IMAP login and webmail interface, both down for an hour and a half. Lots of email probs with DH recently… Not good…

Ours just came back up!

Webmail just came back up but my VPS is still down

Just started doing some major site development and then the site went down. Do i get a refund if I’m losing money or is this something to take to small claims court?

IMAP and webmail back up, but some mail that was sent recently is missing… Wonder if other items…

Our site was down this week and now I cannot get my email. All while running multiple promotions for my business. Change your name to NightmareHost ASAP. These outages are FORCING me to take my business elsewhere!!!

All sorted!! Thanks Dreamhost.

“Site/VPS outages are not related to this issue and you should Contact Support if you are having issues with those services. -dan w” – I did an hour ago and have had no response

I looks like SVN creation does not work as well

My VPS is back online, just after your last update. It seems that the outage is related to the switch

My whole site http://www.liveshotcam.com is deader than the doornail.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver159′ offline due to unscheduled file-system check (RESOLVED)
4:15 pm

Web VPS host server ‘homie-vserver159′ offline due to unscheduled file-system check (RESOLVED)

Posted (March 17th, 2012 at 4:15 pm PST) by Oscar

We apologize for the short notice, but the web VPS host machine, ‘homie-vserver159’ has just been taken offline to complete a file-system integrity check (fsck). The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array anytime this situation arises.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver159′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 17th, 9:34pm PDT: OK, the filesystem check has finshed and the server is back online now. If you notice any problem with your site then please  Contact Support. Thanks! -Dan J

Update Mar 17th, 8:30pm PDT: We’ve just passed the 70% mark on the second pass of the file-system check. -Dan J

Update Mar 17th, 6:05pm PDT: We’ve just passed the 50% mark on the file-system check. -Dan J

This entry was posted on Saturday, March 17th, 2012 at 4:15 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

8 Responses to “Web VPS host server ‘homie-vserver159′ offline due to unscheduled file-system check (RESOLVED)”

“The affected VPS guests can be found in the first comment below this post.”

Yeah sure….

Seriously wondering what the point in a VPS is, $45 a month and it’s far less reliable than the $10 a month shared server I’m on

Affected VPS:

ps23345 ps23431 ps23517 ps23553 ps23705 ps23738 ps24065 ps24252 ps24812 ps24995 ps25289 ps25368 ps25530 ps25767 ps25947 ps25974 ps26008 ps26199 ps26479 ps26961 ps27082 ps27608 ps28474 ps28482 ps28931 ps28965 ps29174 ps29214 ps29314 ps30846 ps36136 ps36221 ps37374 ps37794 ps37795 ps38180 ps38785 ps41391 ps42890 ps43376 ps44150 ps46983 ps50457 ps52021 ps52852 ps54649 ps56843 ps64860

Coming up to 2+ hours downtime and no news on when this will be fixed?!

This is getting ridiculous. For last month or two now my vps has been dropping out for 5 or 10 minutes at a time and now this. And is the online chat ever going to work again? Support now has gone to crap were lucky if we get a ticket answered within 15 hours. Now this. Off four hours so far and that’s after being off for over 15 hours less than a month ago. Why am I paying for a vps. My sites on shared still work.

Can anyone recommend a good hosting company? Ready to move on here.

@Ian: probably a few more hours, there are multiple passes in the FSCK, we’re currently at around 80% on the second of five passes, the passes aren’t all the same length though

@Mike: we’re working hard to answer tickets as fast as possible

for the last 2-3 months dreamhost is going down hill. my ticket history is just blown up! for the first 4-5 months, not a single problem. but now i cant even go a week without sort of drama. please get it together . We had to look for another hosting company for our next project due dreamhost problems. i wonder how much money these down servers has cost us?

Seems to be down again… ps25767

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver20′ moving to new hardware and restoring from backups (RESOLVED)
1:26 pm

Web VPS host server ‘homie-vserver20′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 16th, 2012 at 1:26 pm PST) by brians

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver20′, we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver20′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience! – Brian S

Update Mar 20th, 12:45pm PDT: The issue with the network interface has been corrected, and the  restoration has completed! All VPS guests are online, but if you’re experiencing any issues with your sites, please contact support . – Brandon

Update Mar 20th, 11:50am PDT: The machine is currently unreachable as it’s  experiencing issues with its network interface. Our admin team is working on a resolution and things should be moving forward again shortly. -Brandon

Update Mar 20th, 11:00am PDT: The restore is continuing without any problems, and all VPS guests are currently up and working from our tests. We have noticed a few instances of users with older data, so if you seem to be missing files or are finding outdated files, please contact support so we can help! – Justin

Update Mar 20th, 5:05am PDT: The restore is continuing without any problems, and all VPS guests are currently up and working from our tests. We have noticed a few instances of users with older data, so if you seem to be missing files or are finding outdated files, please contact support so we can help! – Chalise

Update Mar 19th, 8:30pm PDT: The restore is continuing without any problems, and all VPS guests are currently up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! -Dan J

Update Mar 19th, 11:30am PDT: The restore is continuing without any problems, and all VPS guests are currently up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! – Justin

Update Mar 19th, 6:10am PDT: The restore is continuing without any problems, and all VPS guests are currently up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! – Chalise

Update Mar 18th, 9:14pm PDT: The restore is continuing without any problems, and all VPS guests are up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –Seohee

Update Mar 18th, 2:00pm PDT: The restore is going normally, and VPS are up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –Justin

Update Mar 18th, 6:30am PDT: This restore process continues to execute and we’re not showing any broken VPS guests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –JJ G

Update Mar 18th, 1:30am PDT: This restore is still executing and we’re not showing any broken VPS guests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –JJ G

Update Mar 17th, 9:43pm PDT: VPS guests all report as online and the restore process continues to execute. If you run into any problems with a VPS on this host machine please contact support. -Dan J

Update Mar 17th, 4:25pm PDT: VPS guests all report as online and the restore process continues to execute. If you run into any problems with a VPS on this host machine please contact support. – Oscar

Update Mar 17th, 7:05am PDT: VPS guests all report as online and the restore process continues to execute. If you run into any problems with a VPS on this host machine please contact support. -JJ G

Update Mar 17th, 12:18am PDT: VPS guests remain online and the restore continues to process without issue. If you run into any problems with a VPS on this host machine please contact support. -JJ G

Update Mar 16th, 6:56pm PDT: All VPS guests are online, and the restore is continuing without any major issues. Please contact support with any questions! -Brandon

Update Mar 16th, 5:57pm PDT: The issues mounting the hardware have been resolved, and the VPS guests are now coming up one by one. We’re keeping an eye on them to make sure they come back online and start serving the backup data. Once your VPS guest is online, please let us know if you are missing any data. We will gladly assist you in retrieving any missing data from the old hardware. -Brandon

Update Mar 16th, 4:26pm PDT: Our admins ran into an problem while attempting to mount the new hardware to the backup machine. All services are offline at the moment. Our admins are diligently working towards getting the new hardware online. We will continue to post updates as information is made available. Sorry for the inconvenience this has caused. Your continued patience is greatly appreciated. –Brian S

Update Mar 16th, 2:00pm PDT: The backup server has been connected to the new hardware, and the VPS guests are now coming up one by one. We’re keeping an eye them to make sure they come back online and start serving the backup data. The old hardware is still accessible. Please let us know if you are missing any data. We will gladly assist you in retrieving any missing data. –Brian S

This entry was posted on Friday, March 16th, 2012 at 1:26 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

33 Responses to “Web VPS host server ‘homie-vserver20′ moving to new hardware and restoring from backups (RESOLVED)”

ps21281 ps21405 ps21514 ps24627 ps27464 ps30616 ps54102 ps54461 ps5868 ps5876 ps5878 ps5879 ps5881 ps5894 ps5908 ps6191 ps6195 ps6200 ps6214 ps6215 ps6221 ps6228 ps6324 ps6326 ps6329 ps6368 ps6411 ps6765 ps6856 ps7281 ps7469 ps7474 ps8108 ps8190 ps8256 ps8437 ps8740 ps8750 trapper

March 16th, 2012 at 3:30 pmBrandon Sipes Says:

If this is not affecting email, why has my email been bouncing incoming mail for the last 18 hours? I would appreciate customer service responding to my tickets.

This is the mail system at host homiemail-mx8.g.dreamhost.com.

I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.

Dreamhost Email Support

(expanded from
): can’t create user output file. Command output:
procmail: Couldn’t read
“/mnt/jamestiberius/vol/boot/homie/adverb/x9857027/.procmailrc” procmail:
Error while writing to “/var/mail/dhmail”

Final-Recipient: rfc822; bsipes–ariagroup.com@homiemail-mx8.g.dreamhost.com
Original-Recipient: rfc822;bsipes@ariagroup.com
Action: failed
Status: 5.2.0
Diagnostic-Code: x-unix; procmail: Couldn’t read
“/mnt/jamestiberius/vol/boot/homie/adverb/x9857027/.procmailrc” procmail:
Error while writing to “/var/mail/dhmail”

@Brandon, This error is on the mail server, which has nothing to do with your VPS machine. I’ve checked through your support history and it looks like one of our Tech Support Representives has already helped you to resolve the issue.

Panel has been down for about 40 minutes. 8:49 EDT. Went down right as I was helping a new person sign up for an account. Really made me (and Dreamhost) look bad.

Yep, panel down, am just about to meet a client and talk about the ecommerce option… great…paypal it will be then instead…

my VPS is not in the list and i’m presenting so many problems… neeeed heeeeelppp plis… and the chat live?

@Jorge Puebla: What’s your VPS guest and I can check on it for you?

My panel is down and my email service is not working

@Nick A: Is it the control panel at panel.dreamhost.com or some other panel? I checked but see no open ticket from you, and testing a random email address shows no issues. Please submit a ticket with more specific details and we would be happy to check it out for you.

Webmail and email still not working, tickets submitted. Over 12 hours this time and its the second time in three days. Dreamhost is close to losing another customer.

Email is down. Second time in a week. Not happy. No live chat available. Submitted ticket. Ticket # 4824060 I need it fixed asap. I use this as my primary email.

@Brandon & Wes: I have one of our expert support persons looking at your issues now. They should be getting in contact with you shortly.

I have to say that I’m very disappointed in DreamHost this time around. I’ve been through several other episodes with DH and have always found their response prompt and reasonable. But this time is much different.

On Friday, my customer contacted me to note that his sites were down. I filed a support request before there was any notice here on DH Status. As they’ve explained it, hardware failed and was swapped out. Fair enough. Happens.

Then the restore process began. That was *two* days ago. How long can it take to restore a few tens of gigabytes? After 24 hours of no meaningful update (no prognosis setting any sort of expectations whatsoever), I asked for an ETA. That was yesterday. Since then, I’ve received no response whatsoever. Now, over 48 hours into this outage, there is still no information by which any of the customers of this VPS can set reasonable expectations.

I conclude:

1. DH now either doesn’t monitor their HW and waits for customers to alert them to problems OR they don’t care to announce any trouble until their customers notice. Either way: FAIL.

2. DH now doesn’t value the patience of their customers during a crisis and will not provide information by which they can set reasonable expectations. FAIL.

3. DH does not share a shred of the sense of urgency of their customers who run businesses on DH.

The simple questions: What is the magnitude of what must be done? How much of that has been done? What is the projected end time for the restore process? What remains after that, if anything? Why is it proceeding so slowly? Why are answers to these questions not part of the standard status update, here?

three days and we are still down, i pay allot of money for this VPS, you would think it would be up and running by now…

i have sent for support tickets, and nothing, not one reply… need an ETA… can i charge DH for the money i am losing… i should send them an invoice…

3 days of complete downtime here as well, this is absolutely ridiculous

Still no response. And still no idea if/when this ridiculous outage will end. DreamHost support is a ghost town.

If there are any competent staff anywhere near this debacle right now, they’re being betrayed by whoever should be giving clear, substantive status updates. Unbelievable.

Frankly, there should be some serious house cleaning after this. This should affect pocket books. It’s certainly affecting ours.

I completely agree Bruce, what a serious neglect towards paying customers

See the light!
Linode.com

DreamHost, you say to contact you if we are having problems still with machine within the above post… i have put in 4 tickets, and nothing, and your instant chat is down..
—–
Update Mar 18th, 2:00pm PDT: The restore is going normally, and VPS are up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –Justin
——-

Hey Rob, they just updated with this:

Update Mar 18th, 9:14pm PDT: The restore is continuing without any problems, and all VPS guests are up and working from our tests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –Seohee

Yet they can’t spend the time to get to our support tickets. 3 days we’ve been down!!! How long does it take to give us a response or to finish restoring backups?!?!

My email forwarding seems to have gone down as of 6:05 PM EDT March 18, 2012. I can send to my direct email address at verizon, but test emails sent to the forwarded addresses are not going through. My squirrelmail account also seems not to be working, but I don’t really use it, and there are just ten messages there which probablyh came in while I was doing some reconfiguring. I sent a request ticket. Hopefully they will contact me on my direct email which is working.

Now the server is Back, but we got error : “bad_httpd_conf” in all my domains.

@Bruce B: The servers don’t contain just a few tens of GB. Each webserver can contain from 1 to 5TB of data, and must be restored while the VPS are running from a mount to the backup server. Copying too fast, means that the VPS are down during the restore, copying too slow, and the restore takes very long. It is a balance to keep them working while restoring. We do monitor all hardware, but there are gray areas with raid monitoring that we are improving. The restore is nearly complete, with around 1.5 of the 1.8TB of data restored.

@Rob: There isn’t enough info in your comment to look up your ticket and assist you. We are working on getting all open tickets answered as soon as possible, there is a higher volume than normal which is causing some delays in responses.

@Josh: You did not provide enough information for me to look up your ticket and assist you.

@Paul E. Schoen: I’ll see if I can find your ticket.

@Matias Puga: Your VPS is back up, and tickets replied to.

@Justin: thanks for your reply. But I’m not sure what value there is in having the VPS up if none of the sites are up. Why not just take the VPS down and do the fast copy? 70+ hours of downtime is a crippling blow.

At the rate your numbers imply, there are still 14 hours of downtime remaining — just for the restore! That would be an 84+ hour outage! Why should that be considered acceptable? Is there anyone there that thinks that this is a reasonable scenario? That this keeps DreamHost viable in the long term?

I appreciate your reply. But I remain extremely disappointed in DreamHost’s choice of backup/restore strategy, the non-existent customer service, and in the extremely poor status updates over the course of this incident. I’ve been an advocate of DreamHost for years. But that’s come to an end. I think that’s pretty sad.

@Bruce B: It is about the best that can be done to balance a restore from backups while the sites are live. If we were able to detect that this server would have issues further in advance (and we are working on being able to do so), we could copy the data in the background and replace the hardware before it had issues, but after the issues already happen like in this case, there isn’t currently a better solution. It has been an unusually busy start of the year with restores from backups, but these are still very few of the servers that we have, sorry that you happened to be affected by the issues. If you have an issue with your VPS and would like me to look, I can do so if provided a VPS number or other details.

@Justin: Yes. Please look at ps6411. None of the public keys for my shell accounts on that host have been restored. And I’m unable to use the passwords I have on record to SSH to it. All my PHP and Rails sites are still down. And none of my messages to customer support after the first have been replied to (hence my extreme frustration).

@Bruce Sorry to hear of the SSH problems. Unfortunately you must regenerate the keys. With regards to your websites, the apache server is not running. I’ve asked an Admin to take a closer look. We will respond to your support request when we have an solution.

How are we looking now on completion?

@Josh it appears to be very nearly complete, there is about 200GB left to restore.

Are you restoring again ?…

We’re seriously down again………

One thing I do have to admit is that our sites are now much slower in comparison to the old server. Not very impressed with that

The speeds of this new server are disgustingly SLOW

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server homie-vserver211 moved to new hardware and restoring from backups (RESOLVED)
8:45 am

Web VPS host server homie-vserver211 moved to new hardware and restoring from backups (RESOLVED)

Posted (March 15th, 2012 at 8:45 am PST) by Justin@DH

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver211′. The admins are seeing if the server can be repaired via a file-system check, or if a move to new hardware , we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 15th, 11:00pm PDT: We are happy to announce that the restore has completed and the server is now running successfully from its new hardware. If you have any questions, concerns, or issues, please contact support so we can help! – Chalise

Update Mar 15th, 9:39pm PDT: We are showing no issues with any of the VPS guests on this host and the restore is near completion! Please contact support with any questions, or issues! – Seohee

Update Mar 15th, 3:29pm PDT: We are showing no issues with any of the VPS guests on this host and the restore is moving along without any trouble! Please contact support with any questions, or issues! -Brian S

Update Mar 15th, 2:33pm PDT: The backup server has been connected to the new hardware, and the VPS guests are now coming up one by one. We’re keeping an eye them to make sure they come back online and start serving the backup data. We’re also running a file system check on the old hardware to see if it becomes available for data access. We’ll continue updating this post as we see more progress. -Brian S

Update Mar 15th, 1:11pm PST: Our admins are still prepping the new hardware. Services should be restored within the next 30 minutes. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. We will continue to post updates as information is made available. Please check back for updates. – Brian S

Update Mar 15th, 12:05pm PST: We tried replacing the RAID card on the machine, however this did not resolve the filesystem problems. Our admins feel that the best course of action would be to move the machine to new hardware. They are prepping the machine at this very moment. We will continue to post updates as information is made available. Please check back for updates. – Brian S

Update Mar 15th, 12:05pm PST: The RAID card has been replaced and all VPS guests are back online. Thank you for your patience! If you have any further issues, please be sure to contact support Support > Contact Support, in your panel. – Brian S

Update Mar 15th, 11:30am PDT: An admin is currently en route to the datacenter to perform the RAID replacement. All VPS guests on this machine will remain offline until the RAID card is replaced. We anticipate roughly an hour of downtime. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for updates. -Brian S

Update Mar 15th, 10:30am PDT: The admins believe that replacing the raid card in this server may solve the issues that we are seeing with its raid. We apologies for the downtime and appreciate your patience while this hardware swap is performed. We will have more info soon. – Justin

This entry was posted on Thursday, March 15th, 2012 at 8:45 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

10 Responses to “Web VPS host server homie-vserver211 moved to new hardware and restoring from backups (RESOLVED)”

Affected VPS guests:

ps42842 ps42853 ps42878 ps42945 ps43116 ps43133 ps43151 ps43193 ps43409 ps43820 ps43937 ps44005 ps44211 ps44662 ps44670 ps44804 ps44807 ps44814 ps44830 ps44860 ps45357 ps45599 ps48359 ps48768 ps53332 ps54493 ps54496 ps55296 ps56698 ps83450

Will support chat every be re-enabled? It’s been down for months it seems.

Is there anyway to get someone to actually follow up on a support tickets submitted in the panel? I’ve been waiting on simple user account to create for over 12 hours now. It appears that the process is hung.

Thanks.

JD: There are plans to bring it back in the future, but I don’t have an ETA at this time. I found and replied to your ticket.

I’m an not getting any connectivity with my admin site. I cannot enter the site and make up posts to it. Please confirm what is going on asap.

Thanks Justin.

You guys seem to have an extreme number of RAID controller issues/failures. Are you proactively replacing hardware before failure or doing it as whatever RAID controller you use which is obviously defective as it happens? I would hope the former, but I’m led to believe the latter. In which I’m leery of trusting my data with you guys.

My site lost data that I uploaded March 10th. Do you guys really not have a more recent back-up than that? Thankfully I was able to restore from my own backups.

I can’t tell if this is the issue affecting our site, but we still can’t access either our site or our email. Hoping that the critical ticket I submitted gets answered soon!

Goodbye dreamhost. It was a good 6 years, up until the last 4 months. That’s how fickle we customers are. Hello MediaTemple.

@Susan – You are not on a VPS, so this issue wouldn’t be related. I do see that Justin helped you earlier, though, and your ticket has been resolved.

@Mutiny32 – Either/both, depending on circumstances.

@Lori – the backups were from 3/10, so if you uploaded files after the backups ran, unfortunately, they’d need to be re-uploaded.

@Annamarie – To check if you are on a VPS guest from this host, please go to the Users / Manage Users page in your Web Panel and check the ‘machine’ for your user. It looks like you’re having issues with your DNS provider (which is not us), so you would not be affected by this issue. I see you have an open ticket and our techs are in the process of assisting you, though.

@Mike – Best of luck!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver211′ offline for file system check (RESOLVED)
11:59 pm

Web VPS host server ‘homie-vserver211′ offline for file system check (RESOLVED)

Posted (March 13th, 2012 at 11:59 pm PST) by chalise

The web VPS host server ‘homie-vserver211′ is offline for an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We will provide frequent updates below as the check completes. We apologize for the inconvenience that this situation causes!

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please see the first comment below to see a list of affected VPS guest servers.

Update Mar 14th, 2:45am PDT: The file check has completed! Thank you again for your patience while this ran. – Chalise

Update Mar 14th, 1:50am PDT: The file check is continuing to run smoothly! More updates to come as we have them. Thanks again for your patience! – Chalise

This entry was posted on Tuesday, March 13th, 2012 at 11:59 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Web VPS host server ‘homie-vserver211′ offline for file system check (RESOLVED)”

Affected VPS guests:
ps42842 ps42853 ps42878 ps42945 ps43116 ps43133 ps43151 ps43193 ps43409 ps43820 ps43937 ps44005 ps44211 ps44662 ps44670 ps44804 ps44807 ps44814 ps44830 ps44860 ps45357 ps45599 ps48359 ps48768 ps53332 ps54493 ps54496 ps55296 ps56698 ps83450

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)
4:52 pm

Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 13th, 2012 at 4:52 pm PST) by Dan J

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver234′, we are in the process of  moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 17th, 4:50pm PDT: The restore process is now complete. All guests are up at this time. If you run into any problems with a VPS on this host machine or are missing any data then please contact support. -Dan J

Update Mar 17th, 4:30pm PDT: The restore process is close to finishing. All guests are up at this time and the restoring is running without any issues. If you run into any problems with a VPS on this host machine please contact support. – Oscar

Update Mar 17th, 7:05am PDT: The restore process here shows  no problems and VPS guests remain online. If you run into any problems with a VPS on this host machine please contact support. -JJ G

Update Mar 17th, 12:40am PDT: VPS guests remain online and none are reporting as broken. The restore continues to process without issue. If you run into any problems with a VPS on this host machine please contact support. -JJ G

Update Mar 16th, 12:11pm PDT: All VPS guests are online, and the restore is continuing without any major issues. Please contact support with any questions! – Brian S

Update Mar 16th, 5:20am PDT: All VPS guests are online, and the restore is continuing without any major issues. Please contact support with any questions! – Chalise

Update Mar 15th, 9:44pm PDT: All VPS guests are online. The restore is continuing without any major issues. Please contact support with any questions! – Seohee

Update Mar 15th, 4:06pm PDT: All VPS guests are online. The restore is continuing without any major issues. Please contact support with any questions! -Brian S

Update Mar 15th, 12:41pm PDT: All VPS guests are online. The restore is continuing without any major issues. Please contact support with any questions! -Brian S

Update Mar 15th, 10:55am PDT: All VPS guests are online, with the exception of one, which we are working on right now. Otherwise, this restore is continuing without any major issues. Please contact support with any questions! -Brian S

Update Mar 15th, 5:30am PDT: While we continue to work with the one stubborn VPS, the rest of the guests on this server are online! The restore is moving forward without issue, but please contact support with any questions! -Patrick

Update Mar 15th, 12:40am PDT: All VPS guests are online, with the exception of one, which we are working on right now. Otherwise, this restore is continuing without any major issues. Please contact support with any questions! -Patrick

Update Mar 14th, 7:40pm PDT: All VPS guests are online at this time. Backups have been mounted via NFS, while the data restores to the new hardware. If you notice any issues with your VPS, please contact support so we can help! –Dan J

Update Mar 14th, 2:20pm PDT: All VPS guests are online at this time. Backups have been mounted via NFS, while the data restores to the new hardware. If you notice any issues with your VPS, please contact support so we can help! – Oscar

Update Mar 14th, 5:05am PDT: All VPS guests are up and running and the restore process is running smoothly. We will continue posting updates as we have them. If you have any issues, however, please be sure to contact support. – Chalise

Update Mar 13th, 9:24pm PDT: The backup server has been connected to the new hardware, and the VPS guests should all be online now. The restore is running and copying files to the local filesystem. We’ll continue updating this post as we see more progress. –Dan J

Update Mar 13th, 5:56pm PDT: The backup server has been connected to the new hardware, and the VPS guests are now coming up one by one. We’re keeping an eye them to make sure they come back online and start serving the backup data. We’re also running a file system check on the old hardware to see if it becomes available for data access. We’ll continue updating this post as we see more progress. –Dan J

This entry was posted on Tuesday, March 13th, 2012 at 4:52 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

101 Responses to “Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)”

List of servers affected:
ps18424 ps18686 ps21403 ps22719 ps36138 ps36837 ps37718 ps37800 ps49609 ps49641 ps49659 ps49692 ps49791 ps49839 ps49847 ps49874 ps49922 ps50141 ps50178 ps50179 ps50652 ps50653 ps50654 ps50655 ps50656 ps86736 ps86744 ps86766 ps86768 ps86802 ps86806 ps86812 ps86816 ps86959

what the hell does this mean??? we are on VPS ps18686 and were just down for 2 WEEKS! we only got back up a few days ago and now you’re putting us thru the same exact hell again!!! this is impossible!!

does this not mean anything to you people?? you just moved us to new hardware, restored us from backups, and made us wait for 2 weeks!
http://www.dreamhoststatus.com/2012/02/25/web-vps-host-server-homie-vserver234-moving-to-new-hardware-and-restoring-from-backups/

how long do we have to wait for responses to tickets? why is chat permanently offline??

why is our ftp showing outdated files from 2 months ago??? why don’t we have an updated backup considering what the hell we just went thru??? PLEASE put us in touch with Brian H for our support ticket. he was the one who handled our restore last week and knows what HELL we’ve gone thru. at this point there is absolutely no reason why we shouldn’t have our domain monitored & maintained directly by a competent & reliable DH technician. we have been down more than we’ve been up and you have cost us countless amounts of money, time, and resources. we are still trying to recover from the last outage restored only days ago and now we are completely screwed. this level of pure incompetency & repeated failure is beyond words. you are trying VERY hard to put us out of business.

Yes chat has been offline for a long time now , probably months.

support tickets take very long to get answered and yet they are happy to collect our hard earned money.

Dreamhost please respond to ticket #4814876

Just found my problem i think, im on ps29760 , i only have one client and the site is not loading, just says connection timed out. Tried everything from 5 different browsers to clearing caches, cookies, flush dns even in a proxy browser its doing the same.

Stuff this!!! i am supposed to get paid by this client and they are probably checking the site or lack there of, client will call any minute or hour to say the site is not loading. And once again i have to explain that the hosting service is S##T. Time to move on very soon!!!!!!!!!

@Malcolm: your server should be online now, the servers are mounting files from our backups while we copy the files to the local filesystem

@John: ps29760 is not on homie-vserver234, your VPS was not running, I started it up again and it seems to be working properly now

@dan

why was it not running?

really again??? :( we try not to complain when this happens – but sheesh – so often these days.

March 13th, 2012 at 10:09 pmGetting Tired Says:

Down Again?

dreamhost.com down? Know for how long?

Hey look, your *entire* network is down again.

Not again guys. Come on now…

ALL My Dreamhost sites are done, and so is Dreamhost.com. Meh.

Any estimate when the sites will be back up and running?

Dreamhost.com isn’t broken. Everyone knows that the Internet closes at 11:00pm and opens again when you move to another host.

*down. Doublemeh.

Any update on all of dreamhost being down?

All my sites are down. Cannot reach my sites, nor dreamhost.com, nor the webpanel. How the hell am I supposed to deliver these files to my client when they can’t access my site?!?!?!? ARRRRGGGGGHHHHH WHYYYYYYYY WHYYYY WHYYYY

March 13th, 2012 at 10:16 pmJere Krischel Says:

Wow. No web. No dreamhost panel. No dreamhost.com. No webmail. No ssh. No ping to ns1.dreamhost.com.

Nuclear strike?

Everything’s down. The outages have been getting worse. :(

and PLEASE PUT YOUR CONTACT INFORMATION ON THIS PAGE

network-wide outages are so goddamn common – you’d think after the umpteenth time of dreamhost.com itself going down, a way to contact the staff would be put up somewhere that is not the rickety house of cards that is the dh network

Can’t reach my sites, Web panel, or Dreamhost…

I’m getting tired of defending dreamhost. I used to think I was getting a good deal, but I’m starting to rethink my plans.

What on earth is going on here? I thought my internet connection went down, I cannot get to any of my sites nor the control panel.

Looks like this is probably related to the planned maintenance at http://www.dreamhoststatus.com/2012/03/12/scheduled-network-maintenance-in-one-of-our-datacenters/. Hopefully there will be an update there soon.

damnit dreamhost, you’re making it really tough to stay with, i appreciate your amazing prices but there comes a point where the cons outweigh the pros…

Hi guys, can’t contact anyone to find out what’s happened. I was halfway through working on upgrades to a website and it’s cooked itself. Please let us know the estimated timeframe to get this working again. Terribly inconvenient at 4pm on a weekday (we’re not all in the US!).

Typical.

There definitely appears to be more going on at the moment than just the one VPS server – I’m ps22041, so I’m not on that server to the best of my knowledge, but my sites are down right now, and I can’t get through to the web panel, either. Could the process of restoring that server somehow have caused a general network outage?

Granted, it’s 1am where I am, so it’s not exactly prime time, and they only went down around 10 minutes ago, so I’m not about to collapse in weeping hysterics, but there are some updates I’d really wanted to make to a client’s site before I go to bed… *sigh*

Liquidhosting is looking pretty attractive now. My dreamhost loyalty is on thin ice :(

If it’s related to the planned maintenance, my server is not included… maybe someone tripped on the power cable.

my site is down. :((

No websites, no email, no fucking control panel, no support tickets…

Why is my hosting down AGAIN!!!

If you keep taking my business website down my clients will demand I find new hosting. Seriously 2nd time this month!

My site is down, my email is down, dreamhost.net is down, it never ends. And they never mention it in their monthly “news” letters. And they never refund us any money even though there’s a moneyback guarantee.

Our site is down as well as the panel.

Server down. Panel down. Email down. And how can we contact support when all services are down???

ghostrider server is down. dreamhost.com main site as well.

New downtime. Update your blog.

Would this explain why we can’t connect to any databases?
PHPMyAdmin says the db server isn’t responding so that makes sense.
However, the part that doesn’t make sense is the dreamhost panel.

Hopefully this will get resolved before people start asking why their sites are down.

Again Dreamhost Down, loosing traffic…………..

All eight of our sites are down.

Anyone here have a dedicated server and is also experiencing downtime?

kay and saintgeorges are down

The march 12 maintenance notice stated
“We are scheduling an overall network maintenance for one of our datacenters for Tuesday between 10pm and 12am PDT. This will only affect a small subset of our customers. We estimate the actual service interruption for web, mysql, email and the DreamHost panel to be no more than 15 minutes. We apologize for any inconvenience this may cause.”

Not sure how “one of our data centres” and “small subset of our users” turns into a total DreamHost systemwide outtage??? Folks, if you depend on your websites for your business, it’s time to seriously finding another host.

It just doesn’t appear that DreamHost is in control of their network any more. If true, there could be a lot more pain ahead before they get a handle on the situation. I’m all for cutting a hosting company a little slack when things go pear-shaped once in a while, but the past several weeks seem to indicate deeper issues with much longer time horizons to fix.

Argh! Was in middle of working on a site and BOOM it’s all gone. No DH, No panel. None of my sites. I guess Dreamhost says it’s time for bed?

I’m a pretty patient person but this is really getting ridiculous

I cannot believe this. Of all the times for this to happen, I had a very urgent matter with a client that depended on one of my hosted sites and databases, which was to occur tonight. Instead, everything in the project has failed and cost $$$ because once again Dreamhost is dead. I am completely sickened by this. “Our 100% uptime guarantee means we take uptime seriously. -DH” What a load of BS!

jbplayer is down too. Sheesh!

March 13th, 2012 at 10:29 pmGothicCharmSchool Says:

All my sites are down, no email, and so is the webpanel. I’m posting this here because there’s no way to contact support right now.

we’ve started referring to you as nightmarehost.com

Beware the Ides of March…

Looking at other hosting options.

Matt, my thoughts exactly.

Screw it, I’m going to bed. It’s very early here. DreamHost, sort your shit out and at least let us know when you’re going to screw us over so we can plan in advance. Else you’re going to lose a helluva lot of customers. But I guess maybe that’s a good strategy since a lot of problems probably stem from the fact you’re overselling.

dreamhoststatus.com is hosted on linode, for what it’s worth.

Wow this is exciting, it isn’t every day your hosting provider has a 504 error on their site.

starting to think dreamhost might not be the best option for me and all of my websites….

ps18130 is also down?

You idiots are about to lose my business. I’ve been a customer for EIGHT fucking years. I just paid my yearly rebill.
And now, again, my entire site is toast. I’d have changed weeks ago if I had time to deal with the hassle. The *only* reason I’m still with DH is that one simple fact.

I want this fixed, just like everyone else, and like them, I want it fixed *now*. Yeah, right.

Panel is back but can’t access MySQL host or FTP – great client meeting at 9am and I got up at 4am to finisgh this work – this needs soring out quickly please

I’ve been on the Internet for years and you are by far the worst webhost ever. I have never seen anythong like this.

I’m moving on. Two times was enough for me. It’s pathetic, you cause peoples internet experiences to be poor finding sites and you screw your customers but bring negative energy in their life.

You could load you entire client list on another server, just one and deliver better performance. I’ve seen this two times in a month. How these other people stay is beyond me. I’m out, moving on… was worth the couple of bucks I saved.

One of these days your going to get a huge lawsuit filed against you. In fact I find one deal impaired by these outages it will be me.

Michel

General Error
SQL ERROR [ mysqli ]

Can’t connect to MySQL server on ‘db.(removed).com’ (110) [2003]

An sql error occurred while fetching this page. Please contact an administrator if this problem persists.

And so we wait. As do all the people wanting to access our sites.
I once sold a system that consisted of two servers running mirror to each other – one dies, the other takes over.
Yes that would mean doubling up on your hardware. But, guys, the reputation you are losing with these continued outages is hurting you almost as much as it is hurting your customers – us !!
And we still pay our bills so we can have the “service” you guys offer us.

ok back up

Back up, thanks!

Holy hell. What is going on at Dreamhost? I am VERY TIRED of my sites and FTP popping on and off. Week after week, month after month.

“We will post another update shortly with more news. Thanks for your patience!”

1. Last update was over 1 hour and 20 minutes ago. That’s not “shortly”.
2. You are thanking us for our patience. Big assumption, that. We are not patient. Far from it.

OK my sites are up.

Disregard what I previous wrote.

This was all scheduled maintenance as posted as scheduled maintenance.
While I was surprised they would do it so early in the evening in their timezone (mid day for me), at least it was posted up.

It might have helped for something that affects a larger number of users to be stickied to the top before these cluster specific ones.

My website was down this evening for browsing and for FTP transfers, but it just now came back online. The FTP uploads/downloads are now faster than I’ve ever seen it in the last 9 years. So whatever you did, it was a pleasant surprise. Thank you Dreamhost.

Everything seems to be back up now. Might have been back up sooner – I didn’t check between my earlier message and now because I was busy working on something else. So for me at least, everything seems to be working now.

can we please get an update to our ticket (#4815710)? we were told we’d have our files restored last night but they are still not restored.

Hey! Wake up! I know this is not the right place but DO SOMETHING, my tickets are responseless, I have a simple mediawiki to install and it is more than ONE MONTH that dopey server is going down. I really don’t mind how sweet you are, i need that database working.

i need help with my account!!! i only can access to my domains via ssh, please answer the ticket -> #4816072

It´s about 10 hours downtime!!!

My site’s down again. Keeps going down and up, and down and up. Sigh. You should register a new trademark: Yo-Yo-Host.

Our site, kagansblog.com, is down. The error message just says “Database Error – Error Establishing Database Connection”. Any idea when this will be fixed?

Thanks for completely ignoring my support tickets today. Thanks for not letting us know WTF is going on via this blog, too. Come August, after six years using your hosting, I’m moving on. Thanks for nothing.

Dreamhost, Please let us know what is going on (ps27627)

I’m sorry but most of these comments are unrelated to homie-vserver234 so I can’t really address each individually. Most appear to have been related to the maintenance window we had last night.

http://www.dreamhoststatus.com/2012/03/12/scheduled-network-maintenance-in-one-of-our-datacenters/

For the others, if you have not already submitted a support request through the Dreamhost panel I would recommend that you do that. Our support staff will get to each message as quick as possible.

I just send you a ticket #4817901 via dreamhost support panel.

I have no answer for more tan 10 hours, and my all my sites are down for more than 24 hours!!! i give you the answer, there is a problem in my VPS, apache server is not running, you tell me that you have to restart the machine and my /home directory is no to be affected, but now.. i don´t have ANYTHING, all my /home data is lost, and i need it back now!!

what happend to you guys, im hice since 2006, and these is madness…

Please restore the data of my VPS server you are killing my business!!!

Also, you can choose answer one of them: #4817376, #4817354, #4816905

@Jatuv: your VPS is hosted on homie-vserver190, so your issue is unrelated to this post. Your ticket will be handled as soon as possible and note that it is not necessary to send more than one ticket on a single issue. Doing so does not speed our reply, rather it adds to our queue and slows down the process.

@patrickm, thank you for your answer but its the same, my tickets are without answer for more than 8 hours, all my domains are down and i don´t get any solution in any way, here or the dreamhost panel support.

Sorry if this is not the place for this request, but please tell me more about the ticket #4817901 i had lost ALL MY DATA.

Greetings.

Dan J, I am asking from one month a simple thing: someone look at dopey MySQL server, I am unable to use it, looks frozen and slow. One month, do you understand how long this time is? I don’t care if this is a post about a raid-hardware, a backup, a Cisco router, or a blog entry that shows how smart you are (all) in your datacenter.

This is the only way to contact Dreamhost, since my tickets, from move to move, is still unanswered from one week! One week! The chat is gone, what the hell I have to do?

#4818059

ps37081 is down but not listed… WTF?
I’ll be changing hosting if this doesn’t improve!

All of our sites have been down since at least yesterday.
(ps40297) and is NOT listed as an affected site! Please correct this ASAP.
This is costing us $$…again.

@Jatuv: Your ticket was replied to, I am replying now. Your data should be restored.

@Fabrizio: I have reviewed the dopey mysql myself several times, I could not reproduce the problems you were seeing and the server is good. I have already responded to your tickets.

@Bram: Your VPS was hung, I could not get it to respond except to kill it and start it. Unfortunately there was not enough info from the hung processes that I could gather to provide more info.

@Davis: Your VPS should be good now. I am replying to your ticket in a moment.

@Daniel – you told us it would take a few hours to restore our data, but there was a 2 hour delay. it’s been nearly 2 days and we’re still not back so I don’t even see how this short delay factors into the mix. can you PLEASE
update us? we’ve been down for over 2 out of the last 3 weeks!!! this is crazy!

-Malcolm

I’ve contacted support, and got no answer.
ALL my sites are down on ps27627.

Why don’t you tell us what’s really happening, this is NOT a RAID controller failure.

@Malcom Could you please provide us with your VPS guest ( psXXXX )?

@Dan I just restarted your VPS guest. All of your sites are back online

@brians – ps18686. and it’s not that it isn’t UP. it’s that we don’t have our data. the “nightly backup” is 2.5 months old. Daniel J told us we’d have it 2 nights ago! after being down for 2 weeks we were up for 3 business days, still trying to recover from the damage done in the last nearly fatal outage, and now we have the rug pulled out from underneath us again.

also, when will you resume doing nightly backups again or is this no longer happening?

@Malcolm – I see your open ticket and am replying to it directly.

Do this affect my psmysql33707 ?
I can’t reach my Mysql VPS Server since this noon (GMT+7).

@Aphone – no, as noted above, this does not affect MySQL servers, and is only related to the servers listed in the first comment. It looks like your MySQL VPS host machine was rebooted to fix an issue it was having around the time you wrote the comment above and remains stable.

I’m starting to spend more time in these DH forums than I do with friends and family on facebook

@Brian S – can we PLEASE get an update? we’ve been down for 3 out of the last 4 weeks!! when is this rsync going to complete already?

looks like tech responses are in idle mode over the weekend. we’re going on 3 days to wait for the 2nd rsync process. that’s 4 rsync restores in the past month. it seems DH really just doesn’t care. we’re left hanging till no end. how can you let a business site customer be down for nearly 4 weeks straight? and no, please don’t tell us our VPS is UP. it’s useless when our data is missing or corrupt due to rsync failures. this means we are DOWN…STILL

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver186′ moving to new hardware and restoring from backups (RESOLVED)
6:46 am

Web VPS host server ‘homie-vserver186′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 13th, 2012 at 6:46 am PST) by chalise

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver186′, we are in the process of  moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver186′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 20th, 12:45pm PDT: The restoration to the new hardware has completed. All VPS guests are online, but if you’re experiencing any issues with your sites, please contact support . – Brandon

Update Mar 20th, 11:00am PDT: The restore continues to run and should be finishing up very soon. All services appear to be up and running on this server, so please contact support if you are experiencing any issues with your VPS on this server! – Justin

Update Mar 20th, 4:52am PDT: The restore continues to run and should be finishing up very soon. All services appear to be up and running on this server, so please contact support if you are experiencing any issues with your VPS on this server! – Chalise

Update Mar 19th, 7:54pm PDT: The restore is still running, and we’re getting really close to finishing. The machine does still have a slightly higher load than usual, which may be causing slowness for some sites and connections, but our admins are continuing to monitor it, and everything is currently up and running. Please contact support if you are experiencing any issues with your hosting on this host machine! – Dan J

Update Mar 19th, 11:30am PDT: The restore is still running, and we’re getting close to finishing. The machine does still have a slightly higher load than usual, which may be causing slowness for some sites and connections, but our admins are continuing to monitor it, and everything is currently up and running. Please contact support if you are experiencing any issues with your hosting on this host machine! – Justin

Update Mar 19th, 5:35am PDT: The restore is still running, and we’re getting close to finishing. The machine does still have a slightly higher load than usual, which may be causing slowness for some sites and connections, but our admins are continuing to monitor it, and everything is currently up and running. Please contact support if you are experiencing any issues with your hosting on this host machine! – Chalise

Update Mar 18th, 9:16pm PDT: The restore is almost done! We don’t currently have any VPS reporting down, but the machine does appear to have a slightly higher load than usual, but our admins are making sure to monitor it. Please contact support if you are experiencing any issues with your hosting on this host machine! –Seohee

Update Mar 18th, 2:00pm PDT: The restore is getting close to being completed. We don’t currently have any VPS reporting down, but the machine does appear to have a higher load than usual, and we are looking into this. However, please contact support if you are experiencing any issues with your hosting on this host machine! –Justin

Update Mar 18th, 6:30am PDT: This restore process continues to execute and we’re not showing any broken VPS guests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –JJ G

Update Mar 18th, 1:30am PDT: This restore is still executing and we’re not showing any broken VPS guests. However, please contact support if you are experiencing any issues with your hosting on this host machine! –JJ G

Update Mar 17th, 9:35pm PDT: The restore continues to execute smoothly and there are a less than 10 VPS guest restoring at this time. All guests are online now. If you have any problems with a VPS guest please let support know. -Dan J

Update Mar 17th, 4:35pm PDT: The restore continues to execute smoothly and there are a less than 10 VPS guest restoring at this time. All guests are up and running without any issues. If you have any problems with a VPS guest please let support know. – Oscar

Update Mar 17th, 7:00am PDT: The restore continues to execute smoothly, and VPS guest services remain online. If you have any problems with a VPS guest please let support know. – JJ G

Update Mar 17th, 1:10am PDT: All VPS guests are reporting as online and we’re showing no problems with the restore. If you have any problems with a VPS guest please let support know. – JJ G

Update Mar 16th, 12:12pm PDT: The restore is continuing to move along nicely. All VPS guests are online, but please contact support with any questions or issues! – Brian S

Update Mar 16th, 5:17am PDT: The restore is continuing to move along nicely. All VPS guests are online, but please contact support with any questions or issues! – Chalise

Update Mar 15th, 9:43pm PDT: The restore is still continuing smoothly. All VPS guests are online, but please contact support with any questions or issues! – Seohee

Update Mar 15th, 4:12pm PDT: The load is looking much better on this host and the restore is still continuing. All VPS guests are online, but please contact support with any questions or issues! -Brian S

Update Mar 15th, 11:02am PDT: The load is looking much better on this host and the restore is still continuing. All VPS guests are online, but please contact support with any questions or issues! -Brian S

Update Mar 15th, 5:22am PDT: The load is looking much better on this host and the restore is still continuing. All VPS guests are online, but please contact support with any questions or issues! -Patrick

Update Mar 15th, 12:34am PDT: The load is looking much better on this host and the restore is still continuing. All VPS guests are online, but please contact support with any questions or issues! -Patrick

Update Mar 14th, 7:00pm PDT: We’re seeing some high CPU usage on the server at the moment. This is causing slowness on various VPS. Our admin team are trying to locate the cause of that CPU load. All VPS guests are online at this time, though they may be running slower than normal. Backups have been mounted via NFS, while the data restores to the new hardware. If you notice any issues with your VPS, please contact support so we can help! –Dan J

Update Mar 14th, 2:20pm PDT: All VPS guests are online at this time. Backups have been mounted via NFS, while the data restores to the new hardware. If you notice any issues with your VPS, please contact support so we can help! – Oscar

Update Mar 14th, 5:00AM PDT: All VPS guests should now be online, mounted from our backups, while the data restores to the new hardware. If you notice any issues with your VPS, please contact support so we can help! – Chalise

Update Mar 13th, 9:00pm PDT: We’re finishing bringing the VPS guests online on this host. All VPS will be serving data from our back up server while the files are copied to the local file system. There is only a single VPS currently reporting as down and our admin is correcting that now. If you’re seeing any problems with your server then please contact support so we can check it out. Thank you for your patience! -Dan J

Update Mar 13th, 5:52pm PDT: We’re continuing to bring the VPS guests online on this host, and serve data from the backup server.  There are only a handful of VPS still in a broken state and an admin is working to get those up and running so we can start the restore. Thank you for your patience! -Dan J

Update Mar 13th, 3:45pm PDT: We’re continuing to bring the VPS guests online on this host, and serve data from the backup server. Thank you for your patience! – Brian S

Update Mar 13th, 1:20pm PDT: We’re continuing to bring the VPS guests online on this host, and serve data from the backup server. Thank you for your patience! – Brian S

Update Mar 13th, 11:12am PDT: We’re continuing to bring the VPS guests online on this host, and serve data from the backup server. Thank you for your patience! – Andrea

Update Mar 13th, 8:29am PDT: We’re continuing to bring the VPS guests online on this host, and serve data from the backup server. Thank you for your patience! – Andrea

Update Mar 13th, 7:29am PDT: The backup server has been connected to the new hardware, and the VPS guests are now coming up one by one. We’re keeping an eye them to make sure they come back online and start serving the backup data. We’re also running a file system check on the old hardware to see if it becomes available for data access. We’ll continue updating this post as we see more progress. – Andrea

This entry was posted on Tuesday, March 13th, 2012 at 6:46 am and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

68 Responses to “Web VPS host server ‘homie-vserver186′ moving to new hardware and restoring from backups (RESOLVED)”

Affected VPS Guests:
ps27299 ps27353 ps27359 ps27378 ps27398 ps27498 ps27499 ps27510 ps27558 ps27588 ps27627 ps27657 ps27736 ps27765 ps27799 ps27831 ps27961 ps28554 ps28616 ps29316 ps29418 ps29444 ps29455 ps29896 ps30720 ps31144 ps34274 ps35307 ps42191 ps42498 ps45362 ps45374 ps45418 ps45425 ps47656 ps48322 ps48323 ps48324 ps48729 ps48742 ps48801 ps53035 ps53123 ps54513 ps54544 ps54616 ps55630 ps57976 ps57979

March 13th, 2012 at 7:02 amFormer DH Customer Says:

When you get tired of your VPS being down, consider relocating to linode.
http://www.linode.com

Email not working…sigh… checking out Linode?

Squirrel web mail down too…

@anonymical – I just checked webmail on my own email address and it’s loading fine. Please contact support with the details if you’re unable to use your mail service, as it’s not related to this status message. You can submit a ticket through the DreamHost panel, and we’d be happy to help.

I keep trying to submit a ticket but the dreamhost panel and main site are not loading. I am in Germany? Is it loading for you? Also now one of my sites usartberlin.org does not seem to be loading. I am on server Greene….

FTP connection to Greene also keeps Failing…

I have had an urgent ticket in for four hours about email being down. With no reply. No reply to tweets sent to DreamHostCare and no live chat available.

@Andrea, thanks for the response, I would love to post a support ticket but I cannot get into the main site or panel…
@Wes,yes, it seems 4 hours is the time my mail has been down. But now that I see one of my sites also going down I am cringing…

Okay, since I cannot post a support ticket due to Dreamhosts main site being down heres my attempt here….

All sites on Greene are down. Email associated with Greene is down. (except a few who use Cloud Flare http://www.cloudflare.com/)

Heres hoping this gets to someone…. people are indeed dying….at least in a digital spirt way…

@Andrea, last helpful report… .

Using: http://www.iwebtool.com/

I checked dreamhost.com got this report…

HTTP: Website is either loading slow or offline.
FTP: There seems to be a problem connecting to FTP
MAIL (Pop3): There seems to be a problem with the Pop3
MAIL (SMTP): Failed connecting to SMTP
MySQL: The MySQL seems to be offline.

Cannot connect to either ftp server, nor webmail nor dream host homepage since 9am this morning (MET), so there are 7 hours in total now. Since I cannot even access the Panel I cannot issue a ticket. Tweets are not answered. This sucks big time and I lost almost a day of working since I cannot access my mails.
Thank you dream host.

@anonymical – Greene is a web server (does not have email service) and has been up for over 153 days. The panel is online and has been, but if you still can’t submit a ticket, I need you to list at least one site, so I can check on it for you. That will also help me check on whatever issues you may have with your mail, unless you wish to specify an email address.

@Wes – your ticket is being looked into, sorry about the delay.

I experiencing problems with Dreamhost almost one month.

Today one of the accounts I manage just got disabled without warning.

This account is a VPS, but not in the list published above. So far did not respond.

I have a VPS with over 20 websites on it, this is getting ridiculous. What is happening at Dreamhost? I’ve been with them for years and only in the last few months has service plummeted. Whomever is in charge of the data center needs to be replaced. They obviously don’t have a firm understanding of preventive maintenance and customer care.
I was seriously considering upgrading my service with them, but now I may seek another hosting solution.

@Andrea,

Try usartberlin.org

The server status report i get is below. I cannot access the panel yet to create a support ticket.
HTTP: Website is either loading slow or offline.
FTP: There seems to be a problem connecting to FTP
MAIL (Pop3): There seems to be a problem with the Pop3
MAIL (SMTP): Failed connecting to SMTP
MySQL: The MySQL seems to be offline.

Andrea: And while you are at it, could you also try amhass, which is my username on dream host?

@anonymical – your domain is loading, and I was able to log in with the contact@ email address through webmail. The DreamHost panel is online too. It sounds like you may be having issues with your own connection.

@Anna-Maria – your domain is loading, your server has been up for over 28 days, and there are active FTP connections on it. I tested your FTP connection as well and sent you an email with the results.

@Mauricio – please send us a ticket if you haven’t yet about your case. If you already have one, let me know the domain or VPS guest affected, and I’ll look into it for you.

ticket: stemiz1 55318583

So far my site has been down what appears to be six hours during our busiest traffic period. Please look into this for me.

@Andrea, Hmm…thanks for checking it out, but all sites that are not dreamhost that I am going to are working just fine… thats whats really odd. Its only dreamhost sites thats the problem…

On my mail I keep getting this error too…

The server returned the error: The connection to the server “mail.usartberlin.org” on port 995 timed out.

But gmail seems to be just clicking along…

Tried to use an old support ticket to get through…
tracking # is 4813370

@LWE – sorry about that. We’re bringing all VPS guests online, apologies for the delay. Please keep an eye on the status post for updates.

@anonymical – your site is still loading fine. I’m following up on your support ticket now.

Down since 9:00 am EST – 4 1/2 hours. Some better communication would be helpful. This is ridiculous.

At least one of my web sites is down. Is that related to this?

I think these comments and our support tickets are disappearing into a black hole. Sites are still down and this page hasn’t been updated in 4 hours. Any news, Dreamhost?

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee.

As of now you owe me 5 days of Free hosting.

This is really embarrassing, Dreamhost. I’m looking forward to receiving a rebate for the uptime I lost on account of your hardware failures.

@Sam King @WR Could you please provide us with your VPS guest name ( ex. ps12345 )?

@Lester Please check the first comment on this status post. It contains a list of all affected vps guests.

@Dan That’s our word and we are sticking to it. Again, we apologize for any inconvenience this has caused.

i have sites of too many clientes, if u can set priority, its ps29444

WHy can I get but not SEND email?
what’s up?

ps34274 – PLEASE

My VPS server is ps27627 if it helps at all

ps48801
ps48801
ps48801
ps48801
ps48801
ps48801
ps48801

“An error occurred while sending mail. The mail server responded: 4.3.5 Server configuration problem. Please check the message recipient ” and try again.

psmysql28182

My database server is down, don’t know if that has anything to do with this but its not listed on your list.

@Fabio Your VPS is online and site are successfully loading.

@Sam King @Dan @LWE An admin is taking a look at your VPS guests

@tomm9050 Please submit a support request

After six year of happy-dreamhosting I’ve to say that this is a shame. One month after my first support and the dopey MySQL is still slow. I am unable to install MediaWiki for a customer.

AGAIN not being able to send email.. I mean seriously dreamhost this is a joke.

Brian, you said you had an admin looking into my VPS guest, and for a brief while the site worked. It crashed again only 30 minutes later, bringing us to more than 12 hours of downtime. This is frankly unacceptable. FIX THIS

Really, this is a shame. One month of MySQL downtime. This is not professional, feel free to delete my account.

@Fabrizio this post is not related to the ‘dopey’ MySQL server, if you’re seeing slowness on that server you should submit a ticket through the Dreamhost panel

@LWE yes an admin is working on bringing the last few VPS online and fix their various issues

@Chris this post is not related to mail, if you’re unable to send mail from any email address on your account you should submit a ticket through the Dreamhost panel

11 hours now and ps27627 is still down. Not happy…

@Dan: we were able to correct the Apache settings for your VPS and it should be online now

wow again?? more downtime this month, any news of when our sites will be able to operate for more than a couple of days?

@Tracy – Your VPS is not on this host machine. I also checked and your site and VPS are running normally with no apparent issues. I’m also not seeing a support ticket open under that account, but if you need assistance, please submit a support ticket so we can help:
https://panel.dreamhost.com/index.cgi?tree=support.msg&

#Ranting: VPS ps27736 is still not working properly! something seems to be mis-configured! please take note of this. it’s been more than 24 hours already

Our VPS ps34274 is sucking wind – barely responding. PLEASE FIX THIS. We have 3 open tickets, none of which have even been responded too.

@Jeff it looks like the home directory is perhaps not mounting properly, I have an admin checking that out now

@Sam King we just responded to your tickets, it looks like probably the slowness you’re seeing is from the restore process, the files for your vps are currently being mounted from the backup server so that can cause some slowness

@Dan J:
– Server is still not ‘serving’ any data. I’m lacking sleep now waiting this to be fixed. #nothappy

March 14th, 2012 at 7:06 pmdonna or steve Says:

We are not on a vps server but our site is running slow. Anyone else seeing 300+ms on trace routes?

@Jeff the site is working properly for me now, though it is a bit slow

@donna or steve you should submit a support request if you’re getting slow connections to your server

March 14th, 2012 at 8:27 pmdonna or steve Says:

@Dan J i’ve been working with trey – his solution is to change us to a different mysql server?

@donna or steve: Without more info, there isn’t anything I can do for you here. Please submit a ticket if you are seeing issues and we can take a look.

March 15th, 2012 at 10:28 amdonna or steve Says:

@Justin – ticket tracking number is 4818923. been over 24 hours now.

March 15th, 2012 at 10:49 amdonna or steve Says:

@Justin – ticket tracking number is 4818923
I know no one can help but it feels better to share the pain
Lost connection to MySQL server at ‘reading authorization packet’, system error: 0

March 15th, 2012 at 11:14 amdonna or steve Says:

Gateway Time-out
The gateway did not receive a timely response from the upstream server or application.

@donna or steve Your site is back online

March 16th, 2012 at 7:04 amdonna or steve Says:

we are still seeing packet loss, slowness, and errors. See ticket 4819959 for details.

This is getting out of hand. I have been WAY too understanding.
My VPS ps27627 still won’t function properly.
I have rebooted, opened tickets, I’ve increased by memory to 1240MB, and it’s load is in the 600MB’s.
My WordPress sites will not let me into the admin, It times out or gives me a 500 errors.

Day 3 of problems.

The sites come up, then go down. Nothing complex can be done. Static pages work only for a little while. WordPress won’t function properly. I have caching enabled, but admin doesn’t work.

March 17th, 2012 at 4:10 pmRichard Ginn Says:

Same problems as Dan reports above. Plus unhappy clients who want their sites to work.

@Richard: What’s your VPS and I can look into it for you.

March 17th, 2012 at 5:45 pmRichard Ginn Says:

Thanks Oscar!

ps27499

@Richard Ginn: I’ve just replied to your tickets

Re: ps27499

It still takes several minutes to get a WordPress page to come up for editing. I am assuming that DreamHost does have the ability to look and see what is going on, and to fix it. So please do so. Things are NOT OK. Please let us know what you are doing?

Re: ps27499

While working on a WordPress site, and making slow progress, but progress nonetheless, I now get this when I go to back up the site for safe keeping (using the DreamHost WordPress Backup plugin that has always served me well before this current week’s fail):

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@czechcraft.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

So now I’m working without backups :(

re: ps27499

& now the whole WordPress site is offline: http://czechcraft.dreamhosters.com/wp2

Would someone please let me know what is going on? Or not going on? Anything?

@Richard Ginn: I’m checking out your VPS now. Yours is amongst the last to be restored from backups, due to the higher than normal disk usage. It appears to finally be on local disk, which should improve speeds. I’m not sure what the “working without backups” comment means, if you want to clarify in a ticket, I can help with that. We have access to the old hardware and backups, no data should be lost.

March 18th, 2012 at 2:55 pmRichard Ginn Says:

Justin,

I’ve been using a very good wordpress plugin called WordPress Dreamhost Backup for a while now, to backup both the content and the database to the DH backup server. This plugin now fails, with the error message below. So while I have just done a bunch of work on the wordpress site I cannot back it up for safe keeping using this plugin now.

Do you have any suggestions for backing it up now?

The message it now gives is:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@czechcraft.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

@Richard Ginn: that’s a common error message so it’s hard to say exactly why you would be getting that. mostly likely the script is timing out while processing. There are a few reasons it could timeout, one is because of running PHP in FastCGI mode, or it may just be the max_execution_time setting within PHP. You could try setting PHP to run in CGI mode instead and or increasing the max_execution_time setting with a custom php.ini setup on that domain.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Planned maintenance on ‘stampeders’ and ‘michaelkelly’ web servers on Thursday, March 15th @10pm (PDT)
3:04 pm

Planned maintenance on ‘stampeders’ and ‘michaelkelly’ web servers on Thursday, March 15th @10pm (PDT)

Posted (March 12th, 2012 at 3:04 pm PST) by danwy

This Thursday, March 15th @ 10pm (PDT), the ‘stampeders’ and ‘michaelkelly’ servers will be proactively upgraded to new hardware to help prevent any future issues on them. During this time, any sites hosted on these machines will be unavailable. This will not affect email or MySQL services, only web services for domains hosted on these servers. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

We expect the maintenance window to last about an hour and we will be updating this status post with information as the upgrades progress. We’re sorry for any inconvenience this causes you.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you. -Dan

Update Mar 18th 7:30am PDT – This maintenance was completed. No further issues have been detected.

This entry was posted on Monday, March 12th, 2012 at 3:04 pm and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Planned maintenance on ‘stampeders’ and ‘michaelkelly’ web servers on Thursday, March 15th @10pm (PDT)”

My sites are down agaiiiiiiiiiiiiiiiin!!!!!!!!!!!!!!!!!!!!! this is stupid!!! OMG!
Dreamhost sucks !

Got an ETA on completion? Update? Half-completed post in limbo that’s set to go out to 2500 people at midnight. (Ugh.)

My site is functioning very very slowly (some pages don’t pull up at all), I’m on stampeders, is this maintenance still happening and might that explain the slowness? I’ve sent through a support ticket but thought I’d post here too in case you have some info?

thanks in advance!

I am having the same problem as Robert – sites hosted on stampeders are incredibly slow across the board. When can we expect this to change?

My WordPress and Gallery websites have been crawling very slowly for the past few days. A simple wordpress post can take as long as 46 seconds to transfer.

Text-only sites are moderately fast.

Frustrating.

This has been bad ever since stampede was restored from backup earlier this month.

March 23rd, 2012 at 6:56 amEder Acevedo Says:

My sites are working very slowly since last week.

Actually, I’m very tired dreamhost service, there are three issues this year:

1. FTP hacking
2. All sites down a week for a hosting problem.
3. My sites are very slooowly.

Please help me ASAP.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled network maintenance in one of our datacenters on Tuesday, March 13th at 10pm (RESOLVED)
12:01 pm

Scheduled network maintenance in one of our datacenters on Tuesday, March 13th at 10pm (RESOLVED)

Posted (March 12th, 2012 at 12:01 pm PST) by andrea

We are scheduling an overall network maintenance for one of our datacenters for Tuesday between 10pm and 12am PDT. This will only affect a small subset of our customers. We estimate the actual service interruption for web, mysql, email and the DreamHost panel to be no more than 15 minutes. We apologize for any inconvenience this may cause.

Please note, this post is only regarding the upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you. -Andrea

Update Mar 13th, 11:45pm PDT: Network maintenance has completed! Thank you, once again, for your patience! If you notice any issues with your sites or any of your services, please contact support as it is not likely related to this. – Chalise

Update Mar 13th, 11:15pm PDT: Most services should be online at this point. Our network admin is smoothing out a few remaining issues, and we will continue updating with news as we have it until this is resolved. Thanks for your continued patience! – Chalise

Update Mar 13th, 10:45pm PDT: The network maintenance is still ongoing, but we should be wrapping up shortly. Thanks for your patience! – Chalise

Update Mar 13th, 10:00pm PDT: The core reboot in one of our datacenters is currently underway. -Dan J

 

 

This entry was posted on Monday, March 12th, 2012 at 12:01 pm and is filed under Data Center Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

20 Responses to “Scheduled network maintenance in one of our datacenters on Tuesday, March 13th at 10pm (RESOLVED)”

Any update on all of Dreamhost being down?

Site is down for more than 15 minutes, panel is down so can’t submit support ticket

Been down for more than 15 minutes already guys :(

I suppose my site is located out here, I cannot get to squat.

now panel down here. wht is ur time?

Same here. I was just configuring a site, and I guess it might be all messed up. So far, they all seem to be offline on penguin.

Can “always” email support@dreamhost.com or on twitter @DreamHostCare.

I can’t get to any of my Dream-host sites, or Dreamhost.com to file a support ticket. It’s been a lot longer than 15 minutes. Status update, please?

Jeff

Yep, more than 15 mins now guys. Ever heard of email to let people know proactively that you’re going to screw them around again? Yeah, good times.

Hoping this is the reason why Humantorch is down. Unable to submit a ticket due to panel and public site also being down.

If humantorch.dreamhost.com is not affected, then please possibly check this for us. Thanks in advance.

Yea, an email letting me know all my sites were going down would have been nice, since they posted this blog at least 10 hours ago.

ticket submission: normally I reply to an old ticket in my mailbox — to report a new issue is I cant access Panel!

Website and server keeps timing out

Here we go again…

It is all back online for us.

Mine was offline for about 15 to 20 minutes. Back up and running now.

I’m glad they put that clock in the corner of this page.
I usually expect maintenance to be between 2am and 4am pacific so I was surprised that this was right at 2pm in my timezone in the middle of the business day.

However, thanks to that clock that tells me their timezone, it was all not so bad.

Yeah. Down and back up shortly thereafter. Thanks.

Back up, finished the update I was in the middle of. Thanks. Still wish Dreamhost would email when they are going to take your site down.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver127 is running file-system check (RESOLVED)
7:37 am

Web VPS host homie-vserver127 is running file-system check (RESOLVED)

Posted (March 11th, 2012 at 7:37 am PST) by Justin@DH

The web VPS host ‘homie-vserver127 is currently getting a file-system integrity check. This is required due to complications with the RAID on the server which caused switching to read-only mode following a reboot. During this, the host can’t be online, but we’re working on mounting your data from backups, so we can still serve your sites in the meantime. We apologize for the inconvenience that this situation causes! Please contact support if you have any questions!

Please note, this is regarding the VPS host machine ‘homie-vserver142’ and no other services (email, MySQL) or machines are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number. We will continue to update this post as information is made available. Please check back for updates.

Update Mar 11th, 9:00am PDT: The file system check has completed and the server is running again. We will be considering if any further action is needed and will be verifying that the backups are updated. – Justin

Update Mar 11th, 8:00am PDT: The file system check is at 54%. In case of further issues, brand new hardware is being setup for a move to new hardware. We will update this most again when we have more info, the file system check has progressed or we must go ahead with an alternative plan. – Justin

This entry was posted on Sunday, March 11th, 2012 at 7:37 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

6 Responses to “Web VPS host homie-vserver127 is running file-system check (RESOLVED)”

The affected VPS guests are:

ps19707 ps19748 ps19807 ps19854 ps19858 ps20104 ps20823 ps20844 ps21688 ps22003 ps22161 ps22199 ps22216 ps22549 ps23311 ps23652 ps23758 ps23802 ps24183 ps24229 ps25283 ps25733 ps25841 ps28953 ps29921 ps29982 ps29986 ps30330 ps30370 ps30927 ps34717 ps34743 ps34813 ps35070 ps35085 ps35304 ps36090 ps37081 ps37114 ps38327 ps41129 ps42706 ps44783 ps44785 ps46156 ps54413 ps80498 ps82699 ps83949 ps85492 ps85836 ps86688 ps86997 ps89023

Thanks,
how long will be this no service?
10mins, 1hours, 10hours, 1 day?

more or less?

@Ciro: Working on getting this up and working as soon as possible. The file system check appears like it should complete within 1 to 2 hours, if we must go ahead with a move to new hardware, the VPS will be running from the new hardware within a couple hours as well. My apologies if you are affected by this issue.

Great,
the outage was verified 1h30′ ago, it means I’ll wait just few minutes :-)

Thanks for your support.
Ciro

Justin,

Thanks for the update. If there would be a move to new hardware, does that include hard disks? I don’t want to lose any data as we’ve been very active the last past days.

Thanks

@Bram: Generally we would use our backups to restore from, which will lag behind the live server due to amount of data they contain. I most all cases we will have the old hardware available in read-only mode, for restoring any additional data that was not in the backup. If a hardware replacement is needed, once that is confirmed and the vps guests all running, we can restore the data you need via a ticket to support.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘dubhe’ moved to new hardware and restored from backups (RESOLVED)
12:07 pm

Shared web server ‘dubhe’ moved to new hardware and restored from backups (RESOLVED)

Posted (March 10th, 2012 at 12:07 pm PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘dubhe’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘dubhe’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 12th, 4:20pm PDT: This restore is now complete, and all sites are up and resolving. If you notice any issues with your sites or are missing date, please Contact Support. -Brandon

Update Mar 12th, 2:15pm PDT: The rsync on this server has just completed. We’re going to do some post fail over maintenance to make sure everything is ok and we’ll be resolving this status post shortly if we don’t find any remaining issues. Please keep an eye on this post for it to be resolved soon. -dan w

Update Mar 12th, 10:20am PDT: The server continues to remain stable and the restore is still running without any issues. If you notice any problems with your sites or are missing data, please contact support! – Justin

Update Mar 12th, 4:45am PDT: The server continues to remain stable and the restore is still running without any issues. If you notice any problems with your sites or are missing data, please contact support! – Chalise

Update Mar 11th, 7:58pm PDT: The server remains stable and the restore continues to execute without issue. If you notice any problems with your sites or are missing data, please contact support! – Andrea

Update Mar 11th, 12:45pm PDT: The server remains stable and the restore continues to execute without issue. If you notice any problems with your sites or are missing data, please contact support! – Justin

Update Mar 11th, 6:00am PDT: The server remains stable and the restore continues to execute without issue. If you notice any problems with your sites or are missing data, please contact support! – JJ G

Update Mar 10th, 11:00pm PST: The server has remained stable in the last several hours and sites continue to be served as this restore continues to execute. If you notice any issues with your sites or are missing data, please contact support! – JJ G

Update Mar 10th, 5:45pm PST: Some customer sites were not resolving due to the an issue with ipv6 addresses still being bound to the old server hardware. Our system engineering team has resolved the issue and all sites should be resolving and running off mounted backups. If you notice any issues with your sites or are missing data, please contact support! – Oscar

Update Mar 10th, 3:00pm PST: The new hardware is up and all apache instances are restarting. Backups have been mounted and the restore is running. All sites are running from mounted backups. If you notice any issues with your sites or are missing data, please contact support! – Oscar

 

This entry was posted on Saturday, March 10th, 2012 at 12:07 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

10 Responses to “Shared web server ‘dubhe’ moved to new hardware and restored from backups (RESOLVED)”

This is ridiculous. Its like every week with you guys. Can I get a prorated refund for what I paid for a years worth of hosting?

Will a VPS include my uptime? It doesn’t seem like it will based on the underlying hardware issues.

Damn autocorrect.

I meant to ask if a VPS would improve my uptime?

3:00 you said everything was back up. Why is my site still down? Why haven’t you responded to any of my support requests? My site has been down since 7:00am and now its 5:30pm.

@Jeff: We do refund any unused portion of your hosting fees. As for your site being down, I looked for an account under the email address you used to comment and found no account with any hosting services or support tickets. I’ve contacted you through email for a followup.

@John: The VPS service will improve uptime as long as the downtime your experiencing is due to issues related to our shared hosting configuration (such as exceeding a fair share of resource usage on the shared machine). Beyond that VPS mainly offers more isolation from other customers and is one of the primary features, along with the ability to increase memory allocation on the fly.

I’m posting here just because its been 13 hours and haven’t gotten a response from support and I see that @jj replied above helpfully. I’m on taipei – webserver not responding, email and mysql are both responding so its not connectivity. Please investigate!

@Nate: Taipai webserver is not currently having any issues. I found your ticket, your issue should be resolved as of a short time again.

@Justin thanks so much. I got a response shortly thereafter indicating that they had found a firewall problem. All sites came back up for a brief time – I announced to my users, and now they’re all inaccessible again. Perhaps the same problem again?

@Nate: I just helped Scott take another look at your issue, the block should be really removed now.

@Justin – yes, just heard from him and everythings working again. I realize this isn’t an official support channel, glad you were watching it!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘kingston’ moved to new hardware and restoring from backups (RESOLVED)
7:27 am

Shared web server ‘kingston’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 10th, 2012 at 7:27 am PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘kingston’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘kingston’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 18th, 1:15am PDT: This restore has now been completed and the issue has been resolved. However, please contact support if you are experiencing any issues with your hosting on this server! –JJ G

Update Mar 18th, 1:00am PDT: Everything on this machine seems to be running well and the final sync continues to execute. Please contact support if you are experiencing any issues with your hosting on this server! –JJ G

Update Mar 17th, 7:43pm PDT: We are doing a final sync now. The restore process continues to execute and sites are still being served from this machine without issue. Please contact support if you are experiencing any issues with your hosting on this server! –Dan J

Update Mar 17th, 4:35pm PDT: The restore process continues to execute and sites are still being served from this machine without issue. Please contact support if you are experiencing any issues with your hosting on this server! – Oscar

Update Mar 17th, 6:42am PDT: The restore process continues to execute and sites are still being served from this machine without issue. Please contact support if you are experiencing any issues with your hosting on this server! – JJ G

Update Mar 17th, 2:00am PDT: This restore process continues to execute and the services for this machine are running well. Please contact support if you are experiencing any issues with your hosting on this server! – JJ G

Update Mar 16th, 12:15pm PDT: This restore is still finishing up, and all services are currently up and running on this server! Please contact support if you are experiencing any issues with your hosting on this server! – Brian S

Update Mar 16th, 5:10am PDT: This restore is still finishing up, and all services are currently up and running on this server! Please contact support if you are experiencing any issues with your hosting on this server! – Chalise

Update Mar 15th, 9:48pm PDT: This restore is nearing its end and running with no reported issues! Please contact support if you are experiencing any issues with your hosting on this server! – Seohee

Update Mar 15th, 11:15am PDT: This restore is still moving along with no reported issues! Please contact support if you are experiencing any issues with your hosting on this server! -Brian S

Update Mar 15th, 5:19am PDT: This restore is still moving along with no reported issues! Please contact support if you are experiencing any issues with your hosting on this server! -Patrick

Update Mar 15th, 12:27am PDT: We are showing no reported issues with this server and the restore is moving forward. Please contact support if you are experiencing any issues with your hosting on this server! -Patrick

Update Mar 14th,  7:44pm PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please contact support . –Dan J

Update Mar 14th, 2:50pm PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please contact support . – Oscar

Update Mar 14th, 4:40am PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please contact support . – Chalise

Update Mar 13th, 9:30pm PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please  contact support . -Dan J

Update Mar 13th, 5:11pm PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please  contact support . -Dan J

Update Mar 13th, 11:22am PDT: The restore continues with no issues and all services are working properly on the server. We’ve mounted the data for the users who were previously missing, and all sites should be online. If you are experiencing issues with your sites, or still see missing data, please  contact support . -Brandon

Update Mar 13th, 6:24am PDT: The restore continues with no issues and all services are working properly on the server; we’ve found a few users we’ve found that are missing data, which is still being worked on, however, if you notice any issues with your sites or are missing data and have not yet contacted support please be sure to do so! Thanks for your patience. – Chalise

Update Mar 12th, 4:30pm PDT: It has come to our attention that a handful of users on this server are still missing some website data. Our admin team are looking into the matter. We will update this post when we have further information -Brandon

Update Mar 12th, 2:20pm PDT: The restore continues with no issues and all services are working properly on the server. If you notice any issues with your sites or are missing data, please contact support! -dan w

Update Mar 12th, 10:20am PDT: The restore is going well, and currently all services are up. If you notice any issues with your sites or are missing data, please contact support! – Justin

Update Mar 12th, 4:52am PDT: The restore is going well, and currently all services are up. If you notice any issues with your sites or are missing data, please contact support! – Chalise

Update Mar 11th, 7:59pm PDT: The restore is going well, and currently all services are up. If you notice any issues with your sites or are missing data, please contact support! – Andrea

Update Mar 11th, 1:45pm PDT: The restore is going well, and currently all services are up. If you notice any issues with your sites or are missing data, please contact support! – Justin

Update Mar 11th, 6:15am PDT: This restore continues to chug along without issues and the server remains stable. Websites continue to be served and all Apache processes continue to run. If you notice any issues with your sites or are missing data, please contact support! – JJ G

Update Mar 10th, 11:15pm PST: The server has remained stable since the restore has started. Websites continue to be served as this restore continues to execute. If you notice any issues with your sites or are missing data, please contact support! – JJ G

Update Mar 10th, 8:25am PST: The new hardware is up and all apache instances are restarting. Backups have been mounted and the restore is running. All sites are running from mounted backups. If you notice any issues with your sites or are missing data, please contact support! – Oscar

This entry was posted on Saturday, March 10th, 2012 at 7:27 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Shared web server ‘kingston’ moved to new hardware and restoring from backups (RESOLVED)”

Is there some kind of RAID Controller plague in Dreamhost? I mean, really, WTF? Last 20 downtime reasons mostly are RAID Controller related …

@zz: We’ve been replacing problem controllers when we come across them but most of the issues have been drive problems leading to raid failures. That is something we’re currently working on with the use of newer, improved, more reliable drives.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Customer Web Panel is offline (RESOLVED)
5:21 am

Customer Web Panel is offline (RESOLVED)

Posted (March 10th, 2012 at 5:21 am PST) by patrickm

We are very sorry for the inconvenience that this may be causing. The Web Panel is currently offline and our Admins are investigating the issue. While the Web Panel is offline, you will not be able to make changes to your hosting settings. We are working to get this back online and will post updates to the situation below.

This is affecting just the Web Panel and no hosting services whatsoever. All other services are online, so if you are experiencing any issues with your hosting it is unrelated to this issue and you should contact support for assistance when the Web Panel is back online.

Thank you for your patience and check back with this post for updates to this situation.

Update Mar 10th, 6:07am PST: This issue is now resolved! Our Admins were able to get the servers for the Web Panel working correctly again and everything is back to normal. If you are experiencing any issues, please contact support! -Patrick

This entry was posted on Saturday, March 10th, 2012 at 5:21 am and is filed under Control Panel Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

8 Responses to “Customer Web Panel is offline (RESOLVED)”

Like said before. It is not that the web panel is offline. All sites using https are off-line while when you access them via plain http everything seems to be in working other. So the title of this post should be https access problems: web panel off line.

I’ll send the bill for helping you out with this later okay?

@Seraphiel: This outage is completely unrelated to any https issues you may be having with your sites. The Web Panel is hosted in a separate environment from your sites and couldn’t have any effect on your secure hosting. If you are having trouble, please contact support, thanks!

Seems to be back up now…

@patrickm –

If you are having trouble, please contact support, thanks!

Umm…..

With the panel offline, there’s no way for us to contact support. Biggest issue that I have with DH is that there is no way to contact support when the panel is not working. I never have been able to understand why there isn’t a link to contact support that can be done through this page in addition to the links on the panel.

@abu: your issue is unrelated to this post, please use the Support > Contact Support page in your Web Panel for assistance.

You can always email support@dreamhost.com, or hit them up on twitter http://twitter.com/#!/DreamHostCare

@brasscrest: You can contact Support in the following ways:

Support > Contact Support in the Web Panel
Email support@dreamhost.com with your Primary Contact Address
http://dreamhost.com/contact/

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Security improvements and password change reminder
3:43 pm

Security improvements and password change reminder

Posted (March 9th, 2012 at 3:43 pm PST) by kathy

At DreamHost we try to be transparent with our customers when it comes to sharing good news and bad news. Whelp, we’ve got some news to share today, and thankfully for both of us it’s all good!

The web is full of nice people and not-so-nice people. We are on constant watch for the not-so-nice people, and there is no shortage of them. We see thousands of attempted hack attempts against our customer database every day (most are automated), and we have been able to intercept and mitigate them all. Hackers want to touch your junk, and it’s our job to keep them at bay.

Hackers have not compromised any personally identifiable account information and our team continues to work constantly and quickly to be proactive in catching these masked bandits. This notice is an effort to share some things we’ve done recently in our efforts to thwart the ever-present hacker threat.

We’ve upped our already upped-security and taken some additional steps to help protect our customers:

• We performed a full audit of all of our legacy systems, removing unnecessarily stored data and improving the encryption of private data where applicable.

• We put new monitoring systems in place to better prepare us to react to any new threats that may arise.

• We’ve rolled out more thorough auditing processes to make certain our systems are kept in a proper configuration from here on out.

• We want to remind you about our IP-Lockdown feature. When it’s turned on in your Web Panel preferences you’ll be asked for approval if anyone attempts to login to your DreamHost control panel from a new IP address.

• We have adjusted our default FTP and SSH user security settings to prevent tampering by other users.   More information on this setting can be found here: http://wiki.dreamhost.com/Enhanced_User_Security

• We are automatically cleaning hacked sites and proactively looking for signs of compromise. As part of this ongoing effort we are sending diagnostic scans to customers and cleaning certain specific hacked injections that we have found in large quantities.

• We are reminding you to update your CMS scripts.

• We’ve added password strength indicators for all passwords entered in the panel to help make sure your password is super strong.

We encourage you to change passwords to your DreamHost control panel and elsewhere on a regular basis. Might as well do it now!

Below are some best practices to make sure you are extra safe:

• Use different passwords for different accounts.

• Pick secure passwords

• Use our password strength indicator

• Change your passwords regularly

Our security team doesn’t sleep so that you can.

Please enjoy your eight hours free of interruption – on us!

-DreamHost Security Team

This entry was posted on Friday, March 9th, 2012 at 3:43 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

30 Responses to “Security improvements and password change reminder”

Cool, security is good. Thanks.

8 hours of sleep? Wow, great, are you going to come deal with my kids overnight then? :-P

Dear DreamHost Security Team,

Thank you very much!

Sincerely,
A well rested DreamHost customer

So many site hacked tonight … can’t keep up

All my sites were hacked again last night. All of them. Again.

I’ve changed all my passwords to crazy random combinations. I changed all the file permissions as directed. I’ve done everything that Dreamhost asked me to do yet my sites are still being hacked.

What gives? And when will you be cleaning my sites?

And the panel is now offline…

When will it end?

It is not only the panel that is down. This seems to be https related. Non of my sites seem to work when you try to access them securely (https). Plain http works fine. The panel always reverting to https makes it look being down for everybody.

still dealing with hacked sites … it would be nice to have more than 4 hours sleep. getting cranky. sites hacked again. what is the problem? I’ve never had so much trouble. new passwords random random random! files hacked are all php – thousands of them.

thank you.

Thanks!

Much Appreciated!

I recently discovered that my (admittedly simple) website was hacked. Dreamhost tech support were very helpful in identifying some issues (thanks!).

In the end, I moved my entire site aside (mkdir away; mv * away/), most notably the WordPress install that I had neglected to keep up-to-date. I used the web panel to recreate WordPress (since there were several dozen compromised files), provided my existing database password, and waited for the install to complete. One file, LocalSettings.php, I had to take from my original site. I looked at the file and noticed a huge block of encoded text right at the top (no doubt some sort of evil payload designed to be dropped back onto my clean website). Removed that block. WordPress is back online. I’ve set it to auto-apply the latest patches through the web panel.

Over time I’ll move other content as I’ve vetted that it isn’t compromised.

I’ve also implemented a rudimentary intrusion-detection system that emails me any file changes on my account (it runs via crontab). So far, I’ve seen nothing out of the ordinary. Supporting an official intrusion detection system setup would be a great addition to Dreamhost’s offerings.

For those curious, I’ve basically created a mercurial repository, regularly run “find . -ls | sort > files.txt” where files.txt is owned by the said repo. I submit that file to mercurial and a change hook emails me differences (if there are any).

I’m hoping this fixes the WordPress vulnerabilities @ Dreamhost… I operate six WordPress sites, and they’ve been getting hacked & trashed on an almost daily basis, always with the same useless (but prompt!) form email from Dreamhost, advising me to reinstall from scratch, with known good copies of themes & plugins, and to change the user PW, all of which I do, only to have the site hacked again the next day.

I was really beginning to think someone had complete access to the servers, it’s been getting so bad.

This better correct things, or I’m going to have to move my hosting elsewhere, just so I can get some sleep at night!

Yep, us too. Never had a greater invasion of hacker-placed crap in every area of our site like the one we just discovered today. Unbelievable. I’m sure you guys work really hard on protecting us. But, um, this is bad. Really bad.

I loaded my WordPress blog on my HTC phone and I almost showed it to a client today. Good thing I noticed that it was getting redirected to a porn site before showing it to my client! THIS SUX!

It isn’t DreamHost… It’s:

-Your outdated WP software
-Your vulnerable WP plug-ins
-Your DH user account configuration
-The malware et al. on your local computer
-You!

Instead of blaming DreamHost, do yourself a favor, spend some time utilizing the ultimate free support resource that is the Internet, and educate yourself!

I also had 3 websites hacked. A client site, my portfolio and an ecommerce (still in development). It was not good.
One client called me at saturday because his site wasn’t working and I needed a lot of time to realize how to fix that.

I have contacted support and I’m waiting for a response and advice to avoid this, and some care from DH to fix this problem, that seems to reach a lot of customers.

Be very clear: It is Dreamhost – what L says about it being WordPress is not true.

First of all, you need to move your website to a host that utilizes cPanel. I know you like DH but you have to understand that in order to protect their system, they put all your websites in a single home directory – they explained this to me. So every site is compromised as long as one is. It’s because of their crappy software.

And if you know anyone that’s a DH customer chances are you someone (or yourself) that has had a problem with Dreamhost and then you know what a nightmare this company is to deal with. Because they’re using this crazy hosting software that someone wrote and it’s not based on any contemporary security policies and procedures, you know that when you have a problem with Dreamhost it’s going to go on for WEEKS.

But, let me be clear: I had a customer hacked 2 weeks ago. He had a client contact him because his client’s computer has Avast and is reporting malware. DH found nothing. I ran the site through Sucuri, 5 infected files. Cleaned the files. Write DH again, they find nothing again. Ran the site through Sucuri, clean. (Sucuri only checks the one domain.) Avast is still reporting malware. Download the entire website to my Mac and scan it with norton, find 4 infected files, and proceed to delete them. DH writes again, says it still finds nothing. Avast still reporting malware. We scan the site with another website scanner and find the offending javascript that Avast was giving an error on. (It wasn’t my/our Avast, he was getting these very unclear reports from the customer who had Avast.)

Now mind you? The directories on the account that had .old in them – had multiple virus files in them and every single time that Dreamhost gets back to us they say every time that they can’t find anything wrong with the site. And do they reply promptly to a virus report? Absolutely not. At least a day, and one reply came 8 days later.

You could not pick a worse or riskier hosting company than DreamHost, and truthfully it is their customer’s websites that keep hackers interested and busy and they are ruining the internet for everyone. DreamHost’s IP addresses are published, hackers can just set their programs to try every port on every IP address, and never, ever stop.

Having insecurities in WordPress just saves the hackers time – but I assure you nobody is getting hacked on the sites I host or am responsible for, and it’s time that people got real already with DreamHost and their awful service. You can get 24/7/365 service on the phone, chat, email, from domestic employees from Hostgator, Bluehost, or even myself. Look around, there are a million better hosts out there because anyone is better than DreamHost, people.

OH and by the way? THIS IS EXACT BLOG YOU ARE ON RIGHT NOW? This is on WordPress. And this WP installation violates a most simple best practice security policy for WordPress sites. I can see the vulnerability just by revealing the source in my browser.

Is this who you want to be trusting your income, livelihood, family memories, etc to? It’s more than that. When some rogue terrorism outfit hijacks your site and starts plotting world war three from your account, the FBI might believe you and let you out of custody after they interrogate you and lock you up for 6 months. Alarmist? If you watch any news, you know from time to time about stories of arrests of people where they find out someone else was using their unsecured wifi… And while that’s damaging, I can only imagine what people can do if they can host a website under someone else’s name for dishonorable intentions.

rob the computer guy

March 12th, 2012 at 1:50 amBob the Smarter Computer Guy Says:

Rob with all due respect, nearly 100% of these holes (including the ones that were in my sites) were due to crappy buggy plugins with wordpress or outdasted versions of wordpress. If you’re going to fault DH for anything, fault them for giving us a really open and flexible hosting system, so flexible that we can get ourselves into trouble. You can’t point to a single O/S or network weakness that led to this recent rash of problems. It was strictly based on the software people were using and that is THEIR responsibility to maintain.

BTW: the simple solution to keep your customer websites on DH in their own bulkhead so they don’t all site under one user directory is to create different user accounts for each customer site. This way if one client goes bad, they won’t all get affected.

Dreamhost isn’t perfect but you are getting far more than you pay for. Some people might want or need a lot more than DH can deliver and they can and should go out in the market and pay for it. For technically adept folks who know what they are doing, DH is probably the best value on the market.

For people who got hacked, copy that file (rrnuVaccine.php) in the root of your website then execute it from your website. It will clean it pretty well.

Download: https://raw.github.com/walkeralencar/rrnuVaccine/master/rrnuVaccine.php

for info: https://github.com/walkeralencar/rrnuVaccine/wiki/English

After, edit the file .htaccess to remove any unwanted stuff
then do
chmod 555 .htaccess
to make it read only… even for yourself!!!

good website for scan: http://sitecheck.sucuri.net/scanner/

now, Dreamhost… can you scan all the files? and tell us how to protect ourselves?
I would like to know what kind of security I should apply to the files.

Keep up!

:-)

ho… also do this

chmod 555 rrnuVaccine.php

so that file doesn’t get infected again

and personally… I did it to all php files

chmod 555 *.php

Bob, the only simple solution is to move your site away from Dreamhost. Please check your information before you post – because you’re offering factually incorrect information.

No other web host does what DreamHost does with the home directory, and no other web host has the ridiculous rate of infection on Linux based websites that Dreamhost has, plain and simple. When you’re willing to post your website link and credentials, we’ll take what YOU have to say seriously.

There’s no such technical term as it relates to Linux, websites, or WordPress that is called a “bulkhead” and as such, we have no idea what you’re talking about. If it’s a Dreamhost vocabulary word, well that speaks to why everyone needs to start migrating from Dreamhost yesterday. If it’s such a simple solution, then Dreamhost would have their customers configuring their sites that way by default.

Let me be really clear man because you really piss me off with the idea that you think you know why this problem exists: If WordPress was the problem, the rate of infection on DreamHost would be FAR lower than any other webhost, because no other webhost does an incompetent job of updating the WordPress core – they either update everything or nothing.

HostGator, Bluehost and GoDaddy don’t have this widespread problem and why? Because they use industry standard configurations, software and security protocols.

You think Dreamhost offers you something others don’t because you haven’t worked with at least 25 other hosting companies. And if you did, you wouldn’t possibly waste your time with Dreamhost.

However, you missed the point and don’t understand the concept, so don’t try to spread misinformation: All of the websites you have with Dreamhost are in one single home directory so if one site gets infected, they all get infected. That’s what DreamHost explained to me, and that’s exactly what I see. You would be wrong if you were making the claim that making a separate login account would have any beneficial effect on any of this – it doesn’t matter.

And Dreamhost is by far the WORST VALUE on the market. There isn’t any feature in DreamHost that you can’t get with any other web host. In August, September and October 2011, I benchmarked BlueHost, HostGator Shared Hosting, Hostgator VPS, The Planet, and GoDaddy, and Dreamhost was the slowest host 100% of the time. In fairness, they were not that much slower, but every time, they were THE slowest. They never simply served up a website faster than anyone else, they were always at the bottom of the list. And you can only reach technical support via email. Are you just a masochistic computer guy?

Dreamhost uses vocabulary in their control panel that is not industry standard and as such, getting general tech support from anyone but Dreamhost is virtually impossible. For example, mirroring is not a web hosting term, it is a term related to data synchronization. Do you think that having one website come up when you type one of many different domain names is a feature unique to Dreamhost? It’s not. Do you think that configuring your Google Apps mail with DreamHost is some phenomenal feature unique to Dreamhost? It’s not.

Oh and then there’s that wiki that I was told early on to go get documentation from only to find out that I got wrong information from the wiki and then I was told “well that wiki is written and edited by customers, we don’t monitor that.” Oh, smarter computer guy you must be so right – what a value Dreamhost offers: they don’t do their own documentation and sends me to a website that they don’t monitor or edit as their official documentation site. Hours upon hours wasted.

I volunteer from time to time in the WordPress forums and chat room, and the update for 3.3 in mid-December caused numerous problems for people – and for Dreamhost customers, that update was pushed out to their site without prior knowledge or permission. Updates for the .X value are not security updates and are major updates, and should not be blindly pushed out, and for people’s websites to have gone down 10 days before Christmas is a problem.

Dreamhost makes the claim that they keep your WordPress updated. That’s not true. They don’t update your plugins (and they shouldn’t), but if you have a plugin that has a security hole, well, all of you “smarter” computer guys could easily have this get by you and you end up with not only broken websites, but you never check the security because DH told you they were updating your website. So much for that great DH value, eh?

Since it’s customary – and common sense – to check to make sure your plugins are compatible with a new version of WordPress before updating, DH should not be pushing out major, non-security updates in an automated fashion. It’s simply wrong, and you won’t find anyone else advocating the practice.

And those 130 themes you get with that DH install? Check the dates. Most of those themes are years old. Secure? Ha, right. But that’s part of the DH value, right?

Dreamhost makes the claim that they keep your website backed up. Well, I’m not sure about that. The client with a virus on one site asked DH to restore his backup on another site that had also been tampered with, and found out yesterday that the backup only contained 45kb of data, and that they were sorry. I mean they did use the word “sorry” so maybe I shouldn’t be so hard on them considering the value they offer “smarter” tech guys like you, Bob. But back to reality: so much for that great DH value, eh? HostGator and BlueHost have both kept weekly and monthly backups on file where I have used them to restore data for a customer. And when I say weekly and monthly, that means 4 weeks and 1 month, so a total of 5 backups each time. I’ve only had to ask for data to be restored to help someone once on each of those hosts, but each time they were prompt and complete and it solved the problem.

Dreamhost is the only large web hosting outfit that doesn’t offer telephone or chat support as part of their “great value”. Awesome when they don’t follow your “simple” solution and all of your websites get infected for one mishap on one site, right?

I have wasted more time with DreamHost incompetence that I simply will not remain silent on this topic and for every ridiculous post you post, I’ll respond to you point for point. And I’m so incensed about how awful DH is for the entire internet’s safety and security I’m going to up your ante every single time. Because DreamHost lets you think you’re technically adept means nothing, and my 1,000 words about the most basic inferiorities of their web hosting system makes my point several times over.

And since you didn’t automatically know what I was talking about in regard to the blog you are on this very minute, let’s think twice before you have the offensive grandiosity to even jokingly call yourself “smarter” in any way. I may be communicating that by way of listing the endless number of ways in which DreamHost is an inferior web host who is risking the security of the entire internet, but let me be clear that if DreamHost is somehow more secure for reasons I don’t understand, I’m absolutely willing to learn something I don’t know, because I know far from what I would even say is “a lot” about web hosting or the internet. I know as much as I do about these topics because of the zillions of hours wasted by DreamHost while I worked on clients’ websites.

If you found so much as TWO features that you get with DreamHost that you absolutely had to have to run to use WordPress that you know you cannot get through Bluehost or Hostgator for the same price, then post them and if you’re right, I’ll buy you lunch.

And for Flood, your website is not going to continue to operate properly if you change your files to 555 with WordPress, please don’t play with permissions or get some guidance about it. Go to WordPress.org and search for file permissions for the guidelines. (In the event that you aren’t using WordPress, then please disregard.)

Rob,

While I’m not interested in starting a flame war with you, I would like to clear up some of your misinformation.

1.) Don’t attack “Bob” just because he doesn’t list his website. I purposefully avoid listing my website, to keep people like you from annoying me further. But that doesn’t mean I can’t spot your poorly-customized ElegantThemes Webly install.

2.) You are wrong to say things like “you need to move your website to a host that utilizes cPanel”. There have been many, many, many more disclosed vulnerabilities for cPanel than for the DreamHost panel.

3.) Can you provide any semblance of factual evidence to back up your claim that: “No other web host does what DreamHost does with the home directory, and no other web host has the ridiculous rate of infection on Linux based websites that Dreamhost has, plain and simple” ?

4.) If you hate DH and love those other hosting companies so much, and if they are so much better in terms of pricing, security, and speed, then WHY ARE YOU HERE, POSTING COMPLAINTS? How smart can you be?…

5.) Your arguments are all over the place! You are upset because DH offers you 130 free WordPress themes, and some are outdated? So what? I thought that DreamHost was a hosting company, not wordpress.com! You want DH to automatically update your website, but only if it’s a security update, and only after checking your WP code, theme, and plugins to make sure that the update won’t break anything?!? For like $9/month. That is called “being a cheapskate”. You want DH to send you detailed reports every time your website is compromised due to user error, with specific instructions so that you don’t have to think or do any work? Get real!

Now, after reading the last third of your most recent post, I infer that you might be a normal, competent person, who would genuinely be interested in learning and avoiding past mistakes. Instead of being so resistive, do the impossible — assume that you might not know everything, and try to learn… Here are some tips:

-Enable “Enhanced User Security” to protect your home directory! Or don’t use shared hosting! Simple as that!

-Run each website under a separate user account! What’s that?… You want to run all your websites under a single user?… Alright, that’s fine, but if they all become compromised, that is YOUR fault! That’s like using the same password for every single website, and then, once one of those websites gets hacked, and your crappy password is decrypted, don’t cry when it’s used to access every other account you have with that same password.

-Keep your WP installs up to date! Don’t rely on DreamHost to do it for you! If you want to run a website, act like a big boy and take responsibility for updating and maintaining your software. That includes your WordPress software, theme, and plugins. Sure, DreamHost could remove the “One-Click-Install” option (perhaps they should, to prevent people like you from voicing your complaints on the DreamHost Status blog), but they would surely lose customers. Just be glad that you can even get a free software like WordPress, with seemingly limitless themes and plugins, made so that you and others can effortlessly install them. Again, take responsibility for YOUR website and YOUR software!

I’m sorry I’m not understanding something here. This last DH client has multiple user accounts on his DH install. DH tech support said they were all in the same user folder.

When I go to set up a new account in DH, I can see a different home directory… but there is no indication that any of the four types of user accounts would grant permissions to the user to create websites. Now you may understand this, but I’m not sure who else would, because (a) it’s not industry normal terminology to create an FTP user to administer domains and websites, and (b) there is nothing to indicate that if I create a second user, that said user can actually install a domain and a website… that’s not what an ftp or shell account is.

So I think what you’re all telling me is that all of his sites wouldn’t be compromised if he set up other user accounts – although the choice is an FTP or Shell account which has nothing to do with anything related to being an administrator on the DreamHost account. Create a new user account, set up a domain to that user’s home directory and log in with that new user account to then create and maintain the website, with enhanced security, ok. Right?

But wait a minute… even the “main” account can’t view the home directory of the other account? Or can it? Are you saying to not install ANY website on the main account and create user accounts for every single website you want to install?

Then what does one do when they just want to grant someone access to their websites? If my business partner wants to be able to use his own login to the 5 websites I had on DH, I have to create another 5 logins for him so he can get access to the given home directory since enhanced security would otherwise lock him out?

How do I create a login for an employee or designer or anyone else to access the 5 websites with my being able to see their home directory at all times? I think I know the answer but this is COMPLICATED.

And really, how would anyone know this? Because when I click on anything in the help prompts, I get taken to the wiki that, when I was told by DH staff my information was wrong, that the wiki isn’t reviewed by them, so oh well, sometimes it has incorrect information in it. So I’m distrustful and don’t see this as reliable. I’ve gotten more helpful information from screaming my head off on this post than in a year and a half of dealing with DH tech support during client crises.

I still don’t see how any of you think this service is acceptable. Where do you all get this idea that you have some great bargain from these people? No live support whatsoever? What have they done for you that you’re willing to deny yourself this? Have you yet to go to Bluehost or HostGator or Pagely and see that they offer better service for the same price?

I think it’s a big and intimidating project to move a website and anyone would keep their website put at a very high cost. But the risk of dealing with DH is too much.

The point you all seem to miss is that if there is so much hacking going on with DH and they don’t use openly documented software, don’t you believe that the hackers will do more damage sooner or later to the entire DH ecosystem? At least one server? This isn’t worth the risk. How many other sites on your server follow proper security procedures? Likely less than 1%. And do you not also understand that since DH sites are so easily targeted that x% of your bandwidth is dedicated to the DH ecosystem just fending off the hackers’ attempts? DH is so weak on this home directory thing, you probably have 10 times the number of attempts on every website every minute of every day. Maybe that’s the contributing factor to why response times for DH sites were so consistently just-barely-slower than any other web host?

I sincerely thank you for hammering away at me to make that clear, but I don’t really believe that was a relatively easy concept for you to come by in your dealings with DH – the system or the people. Maybe you caught some advice to do that and went ahead and did it and later on you came to see how it worked, but I have yet to see someone’s account on DreamHost that they configured different control panel logins for each website they have. I’ve not seen enough accounts to really make a good judgment about this but I have seen four accounts that had maybe 25 websites total, for clients ranging from 1 to 7 years with Dreamhost. Nobody understood this. And well obviously I’m not sure I do yet until one of you writes back and says if I’m right or wrong. Everyone created a user account for me that was no good for FTP because they all had enhanced security turned on – I always have to log in with the main account name and password.

I am going to pause here. If I’m now understanding this correctly, I’m not sure now if this security isn’t actually more secure than a default cPanel installation, but I just never heard of anything like this before, and it’s fine when there’s one user per website, but I’m interested to hear how to easily give access to anyone else you might be working with?

Thanks in advance. Really.

If passwords are being hacked, get rid of the passwords.

How do people do that. easy, use sftp/ssh with a private public key pair.

Do you have many websites that keep getting hacked. Simple solution, create a new account and spilt the websites from each other.

Everytime you cleanup the site, it gets hack. simple solution, take it off the air. back it up. clean up the files. If you keep it on the air, each time the website is accessed, it will reinfect the other cleaned up files. Once you know its clean, move it back to the live website again.

Use Subversion. This is a godsend for cleaning up hacked sites, this tool allows you to version control your files. If the files are touched, it will tell you, if you don’t like the change, revert them. etc.

Dreamhost allows SSH. its the only way to manage website, you don’t need to have slow cumbersome ftp uploads. ssh allows you to push files so much faster, it allows you to backup and unzip the files. It allows you to view all files that have changed in the last x amount of time. etc.

Dreamhost does not really make any claims to back ups. They allow you to do a full offsite backup, Its well worth it. You do need to pull it off the temp account before its deleted tho. If everyone did offsite backups, these problems might not be such a problem.

If you have gotten infected by the php ridiculousness, this will clean everything up:
———————————————————

for f in `grep “base64_decode” * -rl`; do python ~/clean.py ~/mysuperspecialdomain.com/$f; done

Here’s the contents of the file clean.py:
———————————————————
import sys

if __name__ == ‘__main__':
file = open(sys.argv[1])

data = file.read()
file.close()

pos1 = data.find(”, pos1)

#print data[pos1:pos2]

if data[pos1:pos2].find(‘base64_decode’) != -1:
new_data = “%s%s” % (data[0:pos1], data[pos2 + 2:])

file = open(sys.argv[1], ‘w’)
file.write(new_data)
file.close()

My wordpress sites were hit as well. I noticed it when I got a malware bytes warning on my local pc when I went to check the site, then saw formatting issues on the WP dashboard. In hopes that others will do the same, I am sharing the steps I took to lockdown my sites.

I changed my dreamhost passwords first. Then I updated WP and all plugins then removed plugins. I had many out of date plugins and WP was behind as well. I disabled the “one click install” from Dreamhost on every site to avoid any themes or WP updates coming through that path in case it had been compromised.

Despite that all of that, I still had significant issues. I restored from old backups and fixed one site, but two others were still infected. I ended up signing up for the Sucuri site to monitor and fix my sites. Two of the sites were completely over run with back doors and hacked php files. I tried cleaner_2.4.php and rrnuVaccine.php on the sites and thought I had cleaned everything up. But the next day, the sites were infected again.

I didnt want to spend the money on Sucri, but the infections kept coming back over and over. I looked around for alternatives but none had the high quality reviews that Sucuri did, in addition to the regular monitoring. The money was very well spent.

After that was done (took just a few hours) I decided to take a layered approach to security. I bought the Bullet Proof security plugin (about $40 – http://www.ait-pro.com/aitpro-blog/2835/bulletproof-security-pro/bulletproof-security-pro-features/ ) for all my WP sites. It locks down the .htaccess file, all the main WP files and sets the up to autorestore if they get compromised. It has a ton of other features – including file locking, notifications and php.ini config (which I couldnt get to work with dreamhost but i I did figure out how to create my own phprc file). I added some additional code to the .htaccess files to block known exploits sites and spammers (http://www.wizcrafts.net/exploited-servers-blocklist.html ) I ended up adding in the IP blocks for many other countries (Russian, China, Korea) that have been identified as spammers and hackers. This may end up blocking more people than I want on my one site that is international, so I am monitoring it closely.

I also added the WP antivirus plugin to monitor files changes and I locked down my WP admin with the free Login Lockdown plugin and Lockdown WP. I am experimenting with the WP Firewall plugin now also which monitor all url requests and blocks several different types of attacks.

Finally I use Backup Buddy to backup all of the sites. You can schedule backups and off load then via ftp, Amazon, Rackspace or email. The backup buddy files I had downloaded for one site allowed me to get it back up quickly since little had changed since the last full backup. My other two sites had changed significantly and I had not scheduled backups to runs since (I now have scheduled backups on all sites).

The sites are locked down tight now as near as I can tell. I also deleted all unused plugins and themes, upgraded to PHP 5.3 and added a phprc file to tighten down the sites even more. I have also added the Enhanced user security for dreamhost and added the new IP security for the dreamhost control panel.

All my Sucuri scans have come up negative since i implemented this approach. I check the stats regular on the site to look for evidence of hacks. So far, everything is good. Bulletproof, Wp Antivrus and Sucuri all email automatically if a problem is found. I am keeping my finders crossed.

Was this ever an open status? Or was it posted and closed? This is the first time I’ve seen it a week later. I only found it because I scrolled down to look for something on another subject.

Why do you make some “enhancement” announcements here while others appear at blog.dreamhost.com? It is very confusing to know what to check. Personally I think that you could better intergrate all the customer portal pages into one single community, including blog.dreamhost.com, discussion.dreamhost.com, wiki.dreamhost.com and finally this site dreamhoststatus.com (sure this site is hosted across the country for good reason, but that doesn’t mean it couldn’t be more cohesively linked into a “customer portal”)

I also feel that one additional web panel password enhancement that must be made is the ability for the user to make his/her password expiring. Perhaps some users may not want to use this and they shouldn’t be forced to, but the ability should be there. Perhaps an easy to user dropdown on the “edit profile” page is the best option. never, 1 month, 2 month, …, 6 months.

Users become complacent when not forced to change their password.

I think rob the computer guy protests too much (as well as posts too much for someone who proclaims to hate DH) and it is quite clear that all he is trying to do is to get you to move your sites to his service or to get you to hire him to take care of the things he clearly can’t take care of himself. I had a momentary hack on one of my sites two months ago and after one email to DH I was able to clear it all up in less than an hour. I then did some investigations of my own and found simple chmod to do and a few simple lines of script to add to my .htaccess file that have kept me safe and humming along every since while the rest of the posters here are still being hacked. I agree with “L” keep your sites up to date, do some simple research and if WordPress is your idea of a good site, think again. It is filled with holes, bugs and security risks which most FREE stuff is. If you want a professional site, invest some time and money and stop all of the whining. DH has been great for my web needs with a price and services you can’t get anywhere else. PEACE!

“LakeRat”: This announcement was included in the DreamHost monthly newsletter that was released a week or two ago. Make sure that you’re set to receive monthly newsletters—Go to: DH Panel > Edit Profile > Contact info

Rob: There are many potential solutions you could consider. SSH Keys / ‘Password-less Login’ is a nice solution, and the documentation on the wiki was correct last I checked (http://wiki.dreamhost.com/SSH). Perhaps the better solution would be to learn how to use UNIX Groups (http://wiki.dreamhost.com/Unix_Groups). Is it the easiest thing to do? No. Is the wiki perfect? No. You see….if they make it too easy, or the wiki instructions too simple, a bunch of unqualified people will try it, and then they’ll come here to the DH status blog (instead of submitting support tickets), and b*tch about it. They will rant and rave at the fact that DH doesn’t provide unlimited support for a product that costs less per year than most people spend on bottled water. Imagine all the possible permutations and configurations that one could have on a web server. Now imagine that you have thousands of customers. How could you possibly support each and every configuration? And do so while charging just slightly over $100/year? And do you know the alternative to WordPress? Geocities / Angelfire / Homestead site builder! So it’s important to educate yourself about website security, because you can never depend on someone else to do it for you, unless you are paying them an enormous amount of money, and even then, security can never be assured…

To all the other haters out there… Not to be a jerk, but, if you think you can find a better web hosting for the money – good luck, have fun, and let us know how it goes! And if you think that this web security stuff is easy, just try your hand at running a web server. DreamHost offers VPS and Dedicated hosting. To get you started, here are the first 28-pages (the Introduction) of the Securing Debian Manual: http://www.debian.org/doc/manuals/securing-debian-howto/ch1.en.html

So, Hacked sites abound and this article claims they have “mitigated” all attempts. They WISH. thing is, a bunch of sites have been hacked via a DH hole.. I hate PR people, they just make up their own realities.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server “izar” not accessible ( RESOLVED )
12:52 pm

Shared web server “izar” not accessible ( RESOLVED )

Posted (March 8th, 2012 at 12:52 pm PST) by Oscar

The shared web server ‘izar’, is currently not accessible. Our system engineering team is working on the issue now, and we will update this post once we have more information. We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide.

If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘izar’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 8th, 1:30pm PST:
The machine is back online. All services have been restored. If you notice any problems, please contact support and we will gladly take a look!

This entry was posted on Thursday, March 8th, 2012 at 12:52 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Shared web server “izar” not accessible ( RESOLVED )”

Anything about the massive code injection in php files in some servers?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues to some of our datacenters (RESOLVED)
11:17 am

Connectivity issues to some of our datacenters (RESOLVED)

Posted (March 5th, 2012 at 11:17 am PST) by andrea

We are currently experiencing some connectivity issues to some of our datacenters. Our networking, system engineers and datacenter operations teams have been deployed and are investigating. Sorry about the inconvenience this is causing. Please check back here often, as we will be posting updates as we receive them. – Andrea

Update Mar 5th, 10:30pm PST: As we haven’t seen any more problems related to this issue I’m resolving this post. If you’re having any trouble with your hosting services, it’s most likely unrelated. Please contact support with details, so we can assist you. – Dan J

Update Mar 5th, 6:10pm PST: The DreamHost panel is online again! We are continuing to monitor it, so we’re not resolving this status post just yet. If you’re having any issues with your hosting services, it’s unrelated. Please contact support with details, so we can assist you. – Andrea

Update Mar 5th, 5:25pm PST: The only service still down/intermittent is the DreamHost web panel. We appreciate your patience while working towards resolution. If you’re having other problems, it’s unrelated. You may contact us with the details through Dreamhost.com . Please be as specific as you can, as we can’t reference your account and case without that. – Andrea

Update Mar 5th, 4:15pm PST: We are continuing to look into the issues with the DreamHost panel. Sorry about the continued inconvenience its status is causing. As stated below, everything else is up and running. We have also caught up on mail delivery. – Andrea

Update Mar 5th, 3:15pm PST: As previously mentioned, we’ve done extensive testing and have determined that affected customer services (site, email, mysql, etc) are functional again. While email is working and accepting logins again, this incident has backed up our mail servers and mail delivery is being delayed. All mail sent during this outage should still be delivered once the mail queues get caught up. We’re still also experiencing intermittent issues with our web panel, which our admin team continues to troubleshoot. We’ll update this post shortly with new information as soon as we have it. Please contact support with details, if you’re still having issues with your site services. – dan w

Update Mar 5th, 2:45pm PST: We’ve done extensive testing and have determined that affected customer services (site, email, mysql, etc) are functional again. The DreamHost panel is still experiencing intermittent issues though and our admin team continues to troubleshoot that. We’ll update this post shortly with new information as soon as we have it. Please contact support with details, if you’re still having issues with your site services. – dan w

Update Mar 5th, 2:14pm PST: We’re confident that most issues have been resolved, and we’re cleaning up the remainder problems. Your sites should be up, mail and mysql should be working. The DreamHost panel was just brought back online as well, and we’re monitoring it for stability. Please contact support with details, if you’re still having issues with your hosting services. – Andrea

Update Mar 5th, 1:10pm PST: We’re making progress in bringing back crucial services and are continuing to work on the rest. The DreamHost web panel may be accessible for most of you, but somewhat slow for now. We apologize for the inconvenience this has been causing. – Andrea

Update Mar 5th, 12:26pm PST: We’re continuing to get all services back online. We apologize for the inconvenience this has been causing. – Andrea

Update Mar 5th, 11:50am PST: Correction to the previous update: unfortunately, some of the mysql services (those that are in one specific datacenter) still have sporadic or slow access. Mail services also are more affected than it appeared earlier, but we’re working on correcting that situation as well. The DreamHost panel is still down, but the rest of the services should be accessible. Sorry about the continued inconvenience this is causing. – Andrea

Update Mar 5th, 11:30am PST: We apologize for the short service disruption. Most of our customer data and services are online again, but the DreamHost web panel is not accessible. We’re continuing to investigate the cause and bring everything back up. We appreciate your continued patience. – Andrea

This entry was posted on Monday, March 5th, 2012 at 11:17 am and is filed under General Outages, Major Outage, Network Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

850 Responses to “Connectivity issues to some of our datacenters (RESOLVED)”

no kidding. we are dead in the water. no website, no e-mail, no panel!

March 5th, 2012 at 11:20 amHammer Gaidin Says:

Is this what is causing the outage of panel?

dreamhost.com unavaible. Panel unavailable. http timing out on site.

this is way more than an inconvenience at the moment.

pretty nonchalant message there

You need to post issues like this in 3 minutes on dreamhoststatus.com, not 40.

None of my sites can be accessed via web or FTP. I can’t post a ticket because I can’t get to dreamhost.com.

March 5th, 2012 at 11:22 amAndy Ringsmuth Says:

“Some connectivity issues”?!?!? More like “What used to be DreamHost is currently a smoldering crater in downtown Los Angeles.” That would appear to be more accurate…

All fallen again. It’s a disaster. Last week, 55 hours does not work and some sites are not functioning normally more than 170 hours. Some of my websites with the new update going so slow you can not navigate and therefore are almost inoperative, information has also been lost. This is my undoing

ahhhhhhh come on i was in the middle of coding something….wtf?

I don’t get outages with you often, but when I do… WHOA. :(

Yet again.. you guys need a new datacenter connection really bad

IT’S A GREAT SUNNY DAY NOT MUCH WIND. TO DO LIST : NEED TO BUY MILK, BREAD AND TAMPAX.

“sorry” is the only thing that you always read

time to move on…

Site down !
Damn! Does it really has to happen right now?!
This seems to be the price of cheap hosting…

The panel down…, no support, great !!!!!!!!!!!

Thanks for the update, Andrea. Looking forward to some good news! :)

This is the same problem as usual ….

1. DOS, DNS or Hardware failure causing Network connection between the 2 Dreamhost data centers to fail.

2. VPS and Websites that have their Databases located at the “other” data center are timing out.

3. VPS and Websites that have their Databases located at the “same” data center are working fine.

This seems to be happening over and over and over again with Dreamhost. That and the “dreaded” RAID card failure. Hopefully they get this fixed ASAP (ASAP = 3 hours Dreamhost time).

I can only imagine the hysteria at Dreamhost HQ. Thanks for the update and please let us know an ETA and the resolution.

March 5th, 2012 at 11:25 amUnhappy Customer Says:

Thanks for the update 45 minutes after it happens. Isn’t immediate feedback the point of tools like this?

INCREDIBLE LOW PRICES $$$$ ON V1AGRA, LEV1TRA, BUY NOW AT MED-BARGAINS.LOL
INCREDIBLE LOW PRICES $$$$ ON V1AGRA, LEV1TRA, BUY NOW AT MED-BARGAINS.LOL
INCREDIBLE LOW PRICES $$$$ ON V1AGRA, LEV1TRA, BUY NOW AT MED-BARGAINS.LOL

This has now become a monthly occurrence. Something is going on. I wonder if they are having financial issues.

At least DNS is still working (though latency was terrible earlier).

This is why DH customers should use Google apps or some other third-party email. Their DNS rarely goes entirely down so at least you can still get email during their outages…

This gives me the sads. Good luck in fixing this…soon.

Hey all, its not a datacenter connection issue by all accounts, static pages are being served correctly but anything that requires a DB connection is playing up, ie. email, php etc.

No idea what the issue is though…

Despite all of this I still believe Dreamhost to be a pretty awesome host…. dont like it? Pay more money and go elsewhere, better still, buy your own server and host your own!

zzzzzzzzzzzzzzzzz :)

Somebody’s got a case of the Mondays

ggghhhhkkkkk right when i see a customer email, i can’t get to it. ggghhhhkkkkkkk

some datacenters??? even your own website is down.

Any update on this? It’s really hurting business.

sadface

Does anyone know of other better hosting companies, I can switch over to?

I think someone placed a hex on DH. They have a good stable run for a couple of months and then all of a sudden it becomes a weekly occurrence. This is exactly why we left our previous host. I was on a tech support call every week. DH doesn’t even have a number I can call and give someone HELL. I can’t even get to the web panel to submit a support ticket. This is unacceptable. I understand some downtime, systems fail, it happens — but this weekly business is more than a minor inconvenience.

No no Shadra – it’s soon™ not just soon.

Whats going on?? this is the fourth time this year without EMAILS, FTP, WEBSITES, not even the dreamhost panel to report anything!!
I have a lot of domains hosted with you and i’m starting to worry and looking for better options.
This is unbelievable

For the love of all that is good, will someone give a substantial update on this? I need my site up ASAP!

@Steve
you give me back the money?

Argh… I was about to do some maintenance on my sites, I can log into my VPS but the database is down, the control panel is down.. I see a bunch of people on twitter complaining.

The funniest was: The hamsters that power the @Dreamhost servers are drunk again.

you fools abort now! save your selves… i heard network solutions is much better. is it true?

I really find that incredible that people are having sensible businesses running / hosted on a dreamhost server !?!?!?

there is a price of being a cheap ass…

you want pro, go with a dedicated pro host..

damn…

I’m partly up, but database not connecting to website, so actually down.

Please tell me some hobo wasn’t trying to steal copper.

Reminds me of two nights ago during the tornado reporting…stay tuned to this channel the anchor said…if your power goes out, go to your computer and watch us there…(of course I just put my dogs on a giant power generating treadmill and that creates power for my computer)

Come on Dreamhost, I’m counting on you!

sort it out!!!!!!!!!!!!!!!!!!!!!!

this is taking the pi55

In my 12 years using them, the outages have been quite infrequent, though they have had a few couple of major ones in that time. Have anyone noticed that its always after they change to new hardware here or there. Hope they get their good people to do these upgrades…

@leviatan
I’m sure you’ll live

change pass again :)

zzzzzzzzzzzzzzzzzz

March 5th, 2012 at 11:31 amThe Other Jeremy Says:

well my boss just walked in let see how long till he notices. bets?

no website, no e-mail, and not only that but the web panel is down so I can’t even put in a ticket to find out why & get it fixed.

I’m sick and puking my guts out, I don’t need my livelihood down as well!

When I finally was able to log back into my shell, I found WOP my installs were compromised. Not sure this was “connectivity issues.” Be careful.

no pop mail, no webmail!

please repair it soon as possible.

At least dreamhoststaus.com is always up. Linode must have good network engineers.

Sorry should have read “I found my WP installs were compromised…”

Same shit. Different day. Same crap response.

it’s back up

My sites were down 5 minutes ago, but they’re back now. No idea how long they were down, though.

@RealJeremy: I’m lmfao

============================ME LUV DREAMHOST !!!!!=========================================
============================ME LUV DREAMHOST !!!!!=========================================
============================ME LUV DREAMHOST !!!!!=========================================
============================ME LUV DREAMHOST !!!!!=========================================
============================ME LUV DREAMHOST !!!!!=========================================
============================ME LUV DREAMHOST !!!!!=========================================

operations teams have been deployed means after lunch and a stroll, they will hit reset and see if it works

It’s one thing to be a feature-rich budget host that occasionally has some high profile downtime. It’s another entirely to be down for 40 MINUTES without posting any kind of update or even tweeting to at least acknowledge that you’re aware of the problem! Get on the ball, DH!

my concrete 5 sites are still down, database appears to be compromised

Panel and website is down.

I would gladly pay a few dollars more each month if DH would use it to buy new hardware and drastically improve it’s redundancy.

PLEASE CONVEY MY MESSAGE TO YOUR CEO /PREDIDENT

STOP SELLING HOSTING AND START AS PIZZA DELIVERY BOY, MAY BE YOU ABLE TO DELIVER IN 30 MINUTES….

AGAIN PANEL.DREAMHOST.COM IS DOWN.

DBS AR DOWN

EMAILS ARE DOWN.

JABBER IS DOWNN.

SITES ARE DOWN AS MYSQL HOST IS ALSO DOWN.

WHAT MORE I COULD EVER EXPECT FROM NIGHTMARE HOST.

THERE WAS A TIME WHEN I THOUGHT OF YOUR GUYS MOST COOL.

NOW I FIRMLY BELEIVE YOU ARE PATHETHIC LOOSERS.

ok, it’s back now…whew

I’ve been with DH almost 6 years. Seems every year, I go through a period of rapid fire, short outages that preceeds something major with their system. I’ve had 11 outages in the past 6 days. Nothing longer than a half hour. When I was able to log onto support this morning, they said everything looked fine. Of course. So I just went ahead and requested a server change–the 4th or 5th time in 5 and 1/2 years.
But I am getting a bit tired of defended them and making excuses. I’ll be moving on before my next renewal.

Fix this…all my email clients are down….

panel down, website down, this is the last time. moving sites away today. not sure what happened to Dreamhost, used to be very reliable, for years. now it’s just a nightmare.

My website is up but my email is down AGAIN! This is really too much!

March 5th, 2012 at 11:36 amPatrick Farling Says:

If their entire forward-facing Web presence went down with most of our stuff hosted with them, it’s not likely that they would have immediately noticed the issue unless someone in their office(s) checked their sites, or if there was something affected internally as well. Give them time and they’ll resolve this, as they’re usually able to. Whining and complaining doesn’t help, especially when we could only have informed them sooner if we were able to access the support system.

still down, no web panel, no access. My sites are up, but I can’t access dream host to create and change email addresses.

dh website, panel, webmail, pop down!

I dont have access to “dreamhost.com”, but i can see my websites online ¬¬

March 5th, 2012 at 11:37 amChris Sherwood Says:

It seems our MySQL DB is still down

this is starting to become routine now. WTF is going on over at DH. pretty much ever since they started hiring executives the service is taking a dump!! time to start looking for a competitor to switch to.

ps37343 still down.

Our site is still not connecting to the db

guess I’m one of the few who is still down

Ours goes up log enough to send out 1 email then down again

Re your 11:30 update – do you even bother to check what’s actually running before you post? Sites are still, panel is still unreachable, and even your own home site dreamhost.com won’t’ resolve.

Wow.

NIGHTMARE HOST

STANDING #1 ( LOOSERS)

Booooooooooooo!

A rule of thumb: Don’t hire college interns as network engineers.

Thoughts of the Dreamhost Founders:
“I have an idea! Lets create a hosting company and a shoe string budget. Let’s not worrying about making our hosting fault tolerant. We can just hie behind a crappy support web from when things go wrong. Better yet we can charge extra for a call back support service, but never call them back. LOL, our customers will be a bunch of suckers! You know, one more thing, lets make it really really easy to install crappy canned web site software prone to getting exploited.”

I seriously wonder what the issue is. I’ve been a customer since ’04 and it’s never been this bad. They’ve never been *great*, but it’s gotten much, much worse in the last few months.

Ps6227 Down. Last Week 50 hours down.
Someone webs with fails 160 hours.

March 5th, 2012 at 11:38 amAndy Ringsmuth Says:

I was able to get a little inside information. Not much, but according to one of their techs, “Our core router is unhappy. We’re working on it.”

Although it looks like it’s getting resolved. My mail just started working again, but still can’t ping or pull up panel.dreamhost.com.

@Neo LOL

EXPRESSION OF ANGER IN CAPS LOCK GRRRR ARRGH!

I am entirely sure that the word you were looking for (in the throng of what was no doubt an impressive attack of invisible foes) was “losers”.

ASAP please – I am puzzled that Dreamhost staff isn’t reading these comments

PLEASE CONVEY MY MESSAGE TO YOUR CEO /PREDIDENT

STOP SELLING HOSTING AND START AS PIZZA DELIVERY BOY, MAY BE YOU ABLE TO DELIVER IN 30 MINUTES…….

panel down, e mail down

wp compromised too.

The admin panel is not back yet …
My sites that make MySQL connection that is allocated on a VPS is not returned, but the static (no connection with VPS) have returned …

My site is still down and dreamhost.com is also still down.

March 5th, 2012 at 11:40 amPatrick Farling Says:

@Andy: I’ve got mail connectivity back again, too. Keep your fingers crossed that it all starts coming back soon.

“Most of our customer services are online again.”

Except, you know, the panel. Not important, eh?

Our strict html sites are fine. Only our MySQL database dependent sites are down. Display = “Error establishing a database connection”
We’ve placed a temporary index.html that explains the situation and that loads fine if we rename index.php temporarily.

panel down
all wordpress-sites infected by malware

My site was out for about an hour. (On probably our most important day of the year for us, but whatever.) Glimmer of hope in that it appears to be back up now. Anyone else having luck/non-luck?

Yep, still down here. Can’t get to the panel or any of my email.

My website is up and working fine…however all email accounts are down…Can’t run a business this way. Dreamhost website is down can’t get to panel to register the problem….can’t get to squirrel mail either. Obviously Dreamhost knows about it without me having to tell them. But as other people have commented updates need to be more frequent…40 minutes between updates in the world of internet is 10 years!…

PLEASE get on the ball.

Why do you think wordpress sites are ‘infected’ / ‘compromised’ ?
All I get is a ‘can’t connect to db’ error.

Be careful…the “Error establishing a database connection” is a hack. Check source.

“Update Mar 5th, 11:30am PST: We apologize for the short service disruption. Most of our customer services are online again, but we’re continuing to investigate the cause and bring everything back up. We appreciate your continued patience. – Andrea”

Not accurate…Email is still down.

What are they taling about? “Short”? Short is 2 minutes. Not an hour or more.

Ten minutes after your update that most of your services are online again still can’t get to either the panel or anything using a MySQL database.

I love that I can’t even access the control panel to log an issue. This is the seconds major outage that has taken our corporate site down in as many months. Someone on the top of your network team and disaster recovery teams needs to be fired. Where is the redundancy? Where is the backup plan? Why do I have to pay for a service that isn’t available? I can’t believe that you still have customers with the type of service response you provide. Honestly, I realize that this is cheap hosting, but I’ve had much better experiences elsewhere. Sigh, well that is one less third party hosting I have to worry about.

View source of any PHP file. You will see a BASE64 encoded binary. DO NOT LOAD THESE PAGES IN A BROWSER!

@Ian – would you rather have them work on the panel or on customer websites first? I think they’ve got the order correct – sites first, customer email next, then work on getting the panel working.

All my force to you DreamHost guys. You must be really nervous right now. I know I am.

not again. i dont have access to my sites or dreamhost.com for that matter, and almost a hour latter, thanks for the heads up.

My email is still down. How much longer?

March 5th, 2012 at 11:44 amRussell Peters Says:

I am going to sleep now, in India.

By the time I get up, if my websites are not up, then…

well…

somebody… gonna get a hurt real bad…

somebody…

but I’m not going to say whooo!

NOT REALLY!

STILL THE SAME AS 3 HOURS AGO… NO DATABASE ACCESS!

“Update Mar 5th, 11:30am PST: We apologize for the short service disruption. Most of our customer services are online again, but we’re continuing to investigate the cause and bring everything back up. We appreciate your continued patience. – Andrea”

11:40, still no response to email/webmail. Web site okay (except for “green hosting” image at dreamhost.com).

LOUD NOISES!!!!

Feb 29
Uh huh. Seriously? Dreamhost has been down more in the past 60 days, than ALL of my previous hosts in TEN YEARS!!

From FEB 29 OUTAGE
2:05 pm
Connectivity issues in one of our datacenters
Posted (February 29th, 2012 at 2:05 pm PST) by andrea

Some of our customers may be experiencing connectivity issues right now. This would affect web and mysql services, but only for a small subset of our customers. We apologize for the inconvenience this is causing, and we will be posting updates as soon as we have more information. – Andrea

Update Feb 29th, 2:18pm PST: We are happy to confirm that the situation was quickly identified and has been resolved. There was a slight misconfiguration with our external routing, but it has been corrected. We’re also continuing to investigate to make sure this will not happen again.

this is it…. after last month im outta here after today.

Our sites are up but I can’t edit them in any way.

“Most of our customer services are online again”

Yes, but you’re own website won’t load…

Love you guys, but this is the second to last straw. One more noticeable outage in the next week, and I’m gone.

March 5th, 2012 at 11:46 amNightmareHost Says:

9 red alert issues currently with DuH – really, guys…I use three other hosing companies (in addition to DuH) & their downtime is measured in minutes+seconds, not days and hours

Phew, glad I didn’t actually just destroy my database… I sure hope all the work I’ve done before it crashed wasn’t lost though

php up
html up
web services up
connection to database private server DOWN

Why isn’t “willn’t” a word, I mean seriously?

@dg i got that BASE64 hack, it was the result of timthumb.php.

panel is still down for me.
email is still down.

: (

No email or websites here on the East Coast yet

I don’t envy Dreamhost customer support, but I don’t envy me right now. These outages have probably cost me a contract; it certainly has cost DH at least one.

I have problems accessing to WEBMAIL… my clients are running on circles.

HELP !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I’m losing money… do something please

e mail is still down

March 5th, 2012 at 11:48 amRA Rareminder Says:

I was just fired for recommending you last year. company site was down for over an hour and our clients are not happy. Thank you dreamhost.

this is crazy.

im not back online like their msg says…. no connection to vps database server

site 100% down.

Russell Peters — Nice!

you all need to STFU, computers & networks go in and out, get over it!

This is what you get with a centralized network topology. When one thing goes down, everything goes down.

Hi all — just a check in — D.C. area — email/panel, etc still down — no connection at all.

Thanks for working —

J

Site/MySQL is still down, and game is still txt-bombing me. Lets get the rest of these sites back up and running please! My battery on my poor phone is taking a beating.

Panel is down, email is down, all for the past four hours. WTF?!

This is ridiculous. No site, no email, no panel. Not able to work. Waiting…

e-mail still down on the West Coast!

Yep, everything down for me as well. Been like this for at least an hour. Ridiculous!

Really need to get into the panels area right now. :(

I’m curious how DH will make this up to customers to make them stay.

Way too many outages recently. Paid slightly more with other hosts, and didn’t have as many issues. My clients are frustrated with me. This is a major bummer!

so sick of this.

My websites (which are on the same server) are back up now.

I think this must be the mafian and illuminati messing just like they are messing with me. We’ll see what happens. We’ll see. So that’s how it is.
But but.

still down and out

Website and email are back up. However, webmail is still down and has been for 50 minutes now (at least!)

i heard they hired a new network guy, i bet he broke it, lets all blame him

Now that I can’t even respond to my customers inquiries Dreamhost what do you suggest I say? Offer them a REFUND! Seriously. I have work I need to accomplish. This is happening all to often.

Question to other people in same boat….where are you moving your sites?

As soon as possible, I’m getting off the dreamhost train. Last time I complained of a downtime, they claimed a 100% uptime guarantee. That was two weeks ago.

“We apologize for the short service disruption.” With all respect but this is not a short service disruption.

Error establishing a database connection

http://www.thebbb.org

File a report people. This has just gotten worse and worse as time goes by. I really think they are putting the profits up their nose instead if investing in infrastructure and service.

Everything still offline for me (ds1827) and my tickets are still unanswered. 25+ hours and counting… :(

mail is still down

@jmoney Are saying you were compromised by the timthumb.php vulnerability prior to this outage?

Stating to think it’s time for a walk out. If enough us band together I am sure we can find a host that will make it worth our while to leave.

all my databases are down.

This is absolute rubbish!

Mail stopped working, cannot access panel.dreamhost.com. Great.

So, is it a coincidence that some of my sites have been hacked at the same time as this happens?
All .php files have base64 encoded code inserted at the beginning, and a new /.log directory appears listing a variety of russian urls.

This is it, I’m done giving you guys chances.

Why is it that that weeks can go by without me needing to access the panel, but the minute something happens, and I really need to get in there, it coincides with some major catastrophe?

Again? In 2012 DreamHosta have a DOWN TIME Guiness Wolrd Record!!!

Dreamhost already has an “F” rating with the BBB. Filing additional reports isn’t likely to help matters.

It’s time for another explanation email/blog post from DH management (hey, chance for Simon to shine). This time even dreamhost.com itself is down. Outages have seemed more frequent lately, but something obviously went terribly, terribly wrong here.

Dreamhost hosting/service goes down more than a profitable h00k3r

i have more than 10 site hosted in your machine and many database, a was with off 100% oofline
please help me ….

dofuturo.com.br and all sub domain have many database and all 100% off line
have prevision
this problem are recourred

our email mail.soudermiller.com is still down, can’t ping, can’t get to web panel. our website is up.
i think we are hosted on one of the homie sub5 server clusters
i’m getting pressure from management to change hosts.
Ann

My site is still down. Let us know what is going on!

Oh and panel is still down

March 5th, 2012 at 12:00 pmgreenfishfiles Says:

Seems dreamhoststatus.com is hosted with linode.com (via robtex.com)… Maybe should check them out for hosting. /sarc

website still down ” Data base error”,cannot get into dh website was well.
From Sydney Aus

POP & SMTP – Down
Webmail – Down
FTP – Up
Website – Up

Fix mail now PLEASE.

Still dead in the water. Went black at 10:40 and still out at noon! 80 minutes and counting…

My site is down while I am trying to maintenance it on IP 208.113.223.220 While other site of mine in same DH accounts works on IP 173.236.190.207. kinda of wired!!!!

We can’t get any email in or out… dead in the water. our website is up.. but I need this fixed ASAP. I need to send out some critical items now.

no pop or smtp here… i hate you guys, last time it took days to fix the error

I’m telling you now, this has in some way shape or form, got to be Rob Schneiders fault!

Really disappointed here guys! You’re costing me money.

i’ve been dealing with the php base64 injection for the past month. it’s infecting all php files not just wp. i wrote dh about it and they gave a copy and pasted reply about changing your passwords. i’ve been cleaning out the base64 regularly but the frequency of re-infection has gone from weeks to hours now.

ftp works, 3 of my sites work and the rest are down saying error connecting to database?? any news?

Am I going to be able to recover all the e-mails that I have missed? Have they all been rejected or will I get one big batch when e-mail works again?

My site is up but email and panel are DOWN

Anyone having problems with their databases !!! None of mine work !!!

Webmail – down too
FTP – Up too
Website – Up too

“Error establishing a database connection” for my website

Syd Australia

My sites on vps have database message error… the sites on shared hosting are ok

Pop and smtp down
Webmail – down
FTP – Up
Website – Up

all databases down excluding 3 here, no webmail, ftp works

FTP – ok
sites – slooooow to load but eventually do, wp sites included
email to client – down
webmail – down
dh panel – down

My repository is down!!!!!!!!!!!!

no mail no web mail

March 5th, 2012 at 12:06 pmThe Other Jeremy Says:

I like you dream host. I never thought id have to say it, but i’m getting pressure from the bosses to switch hosting.

placing a call to BLUE HOST and letting them know what’s going on over here. trying for a group discount to switch

March 5th, 2012 at 12:06 pmHumpty Dumpty Owned the Floor Says:

Patient: Doctor Doctor, this steering wheel is stuck in my trousers.
Doctor: What? How did it get stuck there?
Patient: I dunno, but its driving me nuts

I didn’t realize that dreamhost really had an F with the BBB until today. Preferrably you all wouldn’t want me to ever even look at that.

USE IT
SORRY CTRL+C

CTRL+V

website up, email down, squirrelmail still down, no joy here

@YoYo Says: A rule of thumb: Don’t hire college interns as network engineers.

WHO KNOWS IF CEO AND PRESIDENT OF COMPANY IS ALSO COMMUNITY COLLEGE PASSOUT. SO HE HIRED ALL STAFF HAVING EDUCATION AND SKILLS LEVELS LESS THAN HIM.

I’m going to be moving to MediaTemple as soon as possible.

March 5th, 2012 at 12:08 pmThe Other Jeremy Says:

@ sean yeah im going to start checking them out as well.

always liked dreamhost above all, but this with no news is a bit odd….. nothing about issues on their blog either. really need to be able to work here

March 5th, 2012 at 12:08 pmHumpty Dumpty Owned the Floor Says:

oh @neo, you are such a card

Don’t know guys. Been with Dreamhost for a while now, even with the problems, I still think they are a good service. Right now, only the webmail is not working, not sure about my outlook connection to email as I am not at that computer.

I do have a couple of sites on two other hosts out there. If you feel the need to jump ship, you might check them out:

http://www.et-host.com and http://www.magnoliasweets.us/ and they seem to run quite well and have a fair price, even for dedicated servers. While I may use them as well, I personally, will be sticking it out with DH for the majority of my sites.

website still don’t. still can’t access “dreamhost.com”

If you have a WordPress site that can’t get to its database, you can drop an index.html into your root folder to explain the outage to your users (since static pages can be loaded). Here’s what I did:

http://islandforge.com/

You may need to alter your .htaccess file with this line, so the index.html loads before the index.php:

DirectoryIndex index.html index.php

Then, when the problem is fixed, just remove the index.html you made (and restore your .htaccess). This is a quick-fix. Use at your own risk.

website still down. sorry. very sorry

e mail still down

I have client sites that are down, and these are high traffic sites to… Please resolve this issue.

no mail, no panel, n o panel = no ticket = no refund, good job dh

MediaTemple isn’t that much better that what you are experiencing here on DH right now. MediaTemple likes to play games with your bill. not fun. i’d steer away from MT, not worth it.

All email down, websites are showing up, some WP admin panels are loading slowly if at all.

Jorge writes:
> DH doesn’t even have a number I can call and give someone HELL

Haha yeah the psychological side of it is important … but if that’s what you (ab)use support numbers for, you might as well shout out the window, and leave your queue position for someone who can actually get something useful out of support.

… of course, the above only works if there is ANY access …

Did you guys try turning it off and then turning it back on again?

@Sean — Ah, I’ve only heard good things about them. Have any sources? :)

March 5th, 2012 at 12:11 pmLast one out turn out the lights Says:

According to LinkedIn, Simon Anderson took over from the founders as CEO of DreamHost in June 2011. It’s probably no more than a coincidence, but there’s sure been a deterioration in service since then.

IMAP service is still working! For those who need mail service asap, try configure a client with the IMAP service.

problem with mail!!!

IMAP service is still working! For those who need mail service asap, try configure a client with the IMAP service.

@ Luis, what is IMAP service?

The first update confused me as I was like “guess we are not most customers”. Second update at least makes me hopeful in that they recognize the email problem. I can handle the website being down but email down means I am not even getting the complaints to respond to.

down down down
boys boys boys

Panel Still Down for me …

And Database Powered sites on my VPS still down due to DNS issues (can’t connect to database server)

Sales for Blue Host
(888) 401-4678

working on a group discount right now. trying to get a coupon code for us to use

“Fatal error: Uncaught Curl Exception” and also “CURL encountered an error. couldn’t connect to host”
That means Ajax, API and others external services are still down. Please make them back up asap.

Earlier, everything was totally down. Now, my mail servers and Web sites are back up! At this time, my webmail still not working; dreamhost’s web panel still not working.

Give us a free month each time that our sites are down. In last 5 months something like this happen

@dg yes, i’ve had a few sites infected by it, started last fall. Had one that I even updated/patched and cleaned, and just last week it got reinfected again. some nasty nasty stuff going around wordpress sites these days, and it’s not just on dreamhost.

@Kevin:

I understand what you’re saying and I don’t make it a point to abuse Tech Support or Customer Service, however, I think it is important for pressure to be applied from the client base in these kind of events. Having your Tech Support and/or Customer Services phone lines light-up is a far for effective tactic to get them to respond than 217 comments on a blog entry. It’s frustrating as an IT Manager when you have to show face to your bosses of DH’s lack of reliability — especially when you made the case to switch to their service from a previous host.

i experienced more downtime with MT than I have here with DH, but now it’s pretty close.

Update Mar 5th, 11:30am PST: We apologize for the short service disruption. Most of our customer data and services are online again, but the DreamHost web panel is not accessible. We’re continuing to investigate the cause and bring everything back up. We appreciate your continued patience. – Andrea

Andrea what is your qualificaion and experience?

…and everything is back down again…

Everything is still down for me (NYC), including panel and dreamhost website.

Getting a database error.

March 5th, 2012 at 12:15 pmHumpty Dumpty Owned the Floor Says:

hahahahahahahahhahhahahahahahhahahahahhahahahahah

“allbop Says:
@ Luis, what is IMAP service?”

March 5th, 2012 at 12:17 pmHumpty Dumpty Owned the Floor Says:

@neo what ARE your qualifications and experience?

Again!!! This has been happening way to frequently with no explanation, ETA, or corrective action. FYI- the year hosting payment is coming up, Dreamhost better start coming up with an action plan, or I will be transferring my business elsewhere.

Funny thing, dreamhost.com is also down at this time. Hoist on their own petard.

IMAP is a type of mail service like POP3 but the updating (sent and receive) are in real time without working on a offline copy like POP3. The webclient use IMAP. But as web-services are down right now, you have to configure another client with IMAP. Like Outlook or Blackberries. Google it for instructions on how to.

Andrea (and DH):

You really need to stop saying “sorry for the inconvenience”. This is more than an inconvenience – this is an interruption in our businesses and for most, a loss of significant income, credibility, and customer base.

I’m not sure why DH continues to struggle to offer the services that they are contracted to hire – reliable internet hosting – but your team has got to figure this out. Please stabilize and secure your datacenter operations. Moving our services to another host is extremely time-consuming and costly. Nobody wants to do it, but if we can’t count on DH to provide reliable service anything close to your SLA, we’ll have to go.

still not working

We host about 8 sites on our VPS and it’s about half and half as far as what’s up and what’s down. They’re all hosted in the same FTP folder though.

come one…. we need updates… whats going on.

Wrong. My email is configured as IMAP, but it hasn’t been working for over an hour and it’s not working now.

ALL my other sites just went down.

Still cannot access dreamhost panel or website (NYC). There are problems I need to correct and can’t.

IMAP is not working for me.

This is really unfortunate timing and taking way too long to fix.

When will I be able to access my site? I can’t access your site either….this is not good.

imap isn’t workinh on homiemail so there’s no point to switch to imap

@Kevin I’m not looking for someone to pointlessly unload at, don’t get twisted, I just want our systems to run reliably. If I never have to call Tech Support, I’m a happy camper. Remember, I’m an IT professional as well, much like the Tech Support reps and Network Engineers at DH that have to tend to all these matters. I don’t have to hear it from every single employee in our company across all our offices whenever DH’s services go down. It’s frustrating when my office and cell phone ring non-stop and all I can do is submit a support ticket to DH — which I can’t even do at this point because the panel is down. This is really frustrating.

This is becoming the norm. Our clients are NOT happy with us and thus we are NOT happy with Dreamhost. You had better get your act together really fast.
Service was great for years… not it’s down every month. Did you grow too big too fast???

BLUE HOST is looking to get approval for a coupon code for current dreamhost customers to switch over… developing…

@Sean

let me know if you get a coupon code. thx.

My website is still down. c’mon DreamHost.

Just a heads-up to those who are talking about IMAP – mine is still down. This may not be a solution for email currently. I just need one email! :)
With regards to Jorge’s comment about pressure, I am sure they’re feeling plenty of pressure right now from all angles without an extra irate phone call they have to waste their time handling.

@Humpty Dumpty Owned the Floor
Good god you’re so funny!
That’s the kind of answer I’m looking for on a forum!
Have a life!

Maybe you DH guys shouldn’t be tinkering with web servers and such right at the beginning of business on a Monday? Hmm? Just a thought.

Email is still down, some websites are online. MySQL servers still down. No dreamhost.com access, panel still offline. I’m in Oregon on Comcast from multiple physical locations.

thanks for ruining my day dreamhost

This isn’t a connectivity issue. The SQL error is: Too many connections [1040]

Sounds more like DOS type attacks.

Email is still down!! This costs us $$.

Website is up for the moment, but I’d rather have my email up than the website.

Come on DH. Please get your act together.

ok, so, maybe IMAP is another service that is working on some accounts only, like web and ftp. “Sorry for the inconvenience” ;-)

I have at least 15 sites hosted with DH across two accounts and we had a similar experience a couple of years back. The reliability of service was terrible for a couple of months and then it would stabilize only to take a hit again and follow the same cycle. It’s extremely frustrating and “sorry for the inconvenience” doesn’t quite cut it when our several businesses with dozens of employees are just dead in the water.

@Sean and @X_G – I was at BlueHost prior to DreamHost and left after a RASH of HORRIBLE downtime. Days and days of no news, and no service. You’re better off here at Dreamhost.

March 5th, 2012 at 12:23 pmHumpty Dumpty Owned the Floor Says:

@allbop

My life is websites, but they are all down.. so all I have is you, all I wanted was your love!

March 5th, 2012 at 12:23 pmin the worst moment Says:

another non happy customer

Our website works, but no email access.

My sites are down… Please fix it

This is getting a bit ridiculous… an outage for this long?

Cloudflare has saved one of my websites.

BLUE HOST is going to set up a special link for us to provide a special limited time only to switch… waiting for the link…

i was able to access the database 10 hours ago. I can’t get to anything else there right now either. i don’t think i CAN change anything with the dreamhost site itself being down.

“…but the rest of the services should be accessible”

They are not. DH, Please post more frequent updates.

Oooppsss. Take it back. Our website is now down as well.

Email down, panel down…my clients can’t wait.. “Please check back here often, as we will be posting updates as we receive them” WHERE are the UPDATES ?

March 5th, 2012 at 12:26 pmCraftyLittleHobbit Says:

@dtmp

I was looking at Cloudflare. Have you got all your sites signed up or just the one that got saved?

@RealJeremy I’m sure they are (or one would hope) and I don’t envy those Network Engineers at DH in the least, however, they need the boot to their neck from their clients because we have to make explanations on our end as people who have to answer to our users, our executives and our customers for their lack of stability and reliability — especially 2012 YTD.

Bla.. Bla.. Bla… Sorry.. Bla bla bla.. but it´s getting ridiculous!!!!

My FTP is Up. Mysql servers still Down. Please, resolve!

Unreal. The amount of business and embarrassment our brand is undergoing at the moment feels incalculable.
We pay for VPS, MYSQL, and dedicated Memory, on an account since 2005 – do we seriously need to consider other alternatives?? Can anyone recommend a “reliable” host?

@Sean

I haven’t left.. waiting eagerly for link. Believe it or not. I’m not the owner of the site that’s down. I’m actualy a customer waiting to give the link to the site owner.

Again DOWN!! I cannot believe this.. this is a joke. I will have to resort to the BBB. I am sick and tired of this BS. Constantly in fear of emails and websites not working.
I am hosting over 10 websites and now dreamhost is exposing my company in the worst light.. Unbelievable

lost a few clients this morning guys! not smiling over here. any news?

I have been able to ftp using Transmit on my mac

AHHH no email…

All websites are up. Panel is down. http://www.dreamhost.com itself is down!

March 5th, 2012 at 12:28 pm7yr Customer Says:

Has it come to this…
I have been with dreamhost for 7yrs today is the first that I have looked at other hosting services.

http://www.godaddy.com/hosting/web-hosting.aspx?isc=hos3gca10&ci=9009

I feel very upset and angry that you are pushing all the loyal customers away.

Resolve and offer 1 months free hosting or something except the same old responses.

No email here, no web panel access…

HALP!

websites are NOT up. there is no connection to DB for my private servers.

Can someone please give me an alternative to this joke?

Seems everything on dreamhost has exploded. All my websites offline and dreamhost.com unavailable. http://techie-buzz.com/webmaster-tips/dreamhost-down.html

The zombie apocalypse has begun at dreamhost!

It can not be out again. waste time and money when it will reset? I need an answer ….

Odd. My sites are fine. It’s just my (IMAP) mail that is out. Not sure how that can be…

Again DOWN!! I cannot believe this.. :(

I have no email

i cant access email for several hours now. cmon.

BLUE HOST is gonna hook us up. Sales representative JOSEPH at BLUE HOST is setting up a link that we can use as a special offer (time sensitive) to switch. The link is in the process of being set up now… developing…

Panel is still down for me. But at least my web sites are up.

Still not. And from the amount of Twitter posts from other users, I’d say the majority are offline, not restored.

OBsIV are you here? Please join the IRC.

Andrea- thanks for telling paying customers nothing. CAUSE, ETA, RESOLUTION, PREVENTION????

Also still down and y’all refuse to post a tech support number.

I highly suggest Dreamhost gets with the program ASAP, or I will be moving my hosting elsewhere.

dreamhost… you suck!

I guess they don’t run the dreamhoststatus.com site on their own servers, lol. Hey DH, tell us who hosts this site so we can all switch.

really looking forward to the March edition of the Happy Dreamhost Newsletter 8-/

as they begin to “fix” this issue, i find myself here. now without a website that’s down that wasn’t before when everyone else was having a problem!

curious about “Blue Host” as Sean posted about above; will they be any better than this though? Thats my quandary.

site is still down…

email was up for two minutes… now down.

My contract expires at the end of the month and I trying to find one reason to stay with DH.

OOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOO
OOOOOOOOOOO
OOOOOO
OO

Andrea, I have multiple sites still down. I just want to make sure you know since I have no other way to tell you….

I dont want to go for a dip on Sage Weil’s spa; would likely be out of water

Completely black for last two hours. Smack in the middle of prime time. This is killing us!!!!!!

waiting for link from Sean for blue host

I think I’ll cancel this shit server, Dreamhost, me back my money!

hardly a mansion, not even a great location…

March 5th, 2012 at 12:37 pmJaime Linhares Says:

Eu preciso mandar emails!!! I Need Send Emails!!!!!

This really is rubbish. Our world.edu database is down yet again

Considering a VPS is $15/month + $15 for the mysql, $30 for 300Mb of Ram is not a value price for what you get. Dreamhost needs to change their slider prices for VPS.

Also, I tried the domain restore last night for the first time, failed instantly.. lost a lot of data.. Why have a feature and advertise it if its broken? Every other host does backups..

FIX THIS SHIT NOW!

March 5th, 2012 at 12:39 pmCraftyLittleHobbit Says:

F5

anyone know if emails are being bounced, our am I at least going to see them all when this is back up and running?

1-714-706-4182

Is this DH’s phone number?

Still no email…

Umm.. guys? No offense, but you’re getting unlimited everything for less than $10 a month. I’m hosing over TWENTY sites – that’s FIFTY CENTS a month for a website.

If you can’t handle being down for a couple hours a month/year/whatever, and it’s that much lost revenue to you, get an account with rackspace! They’ll be glad to accomodate you, and charge you accordingly.

Do you really think that dreamhost is not on full alert about this right now? Will it really help resolve things if you guys light up their phones and keep techs busy while you complain about something they can’t do anything about?

Oh, and $2 million in that area is hardly a mansion. Check the street view.

Yeesh,
d.i.

“Most” sites may be back up, but my VPS is still down.

You guys are really bad at this…

Doug

Hi everyone,

I’m also facing the same pain like you, that’s why I take a two prong approach.
I host few websites on dreamhost (mostly marketing material – static portion) but my income-producing portion (database and dynamic web pages) I host them on Heroku (www.heroku.com) which is backed by Amazon’s EC services and I’ve only had one outage in more than 3 years of running with them.

If any of you is interested in implementing such strategy do send me a note and I’ll be happy to help you and guide you on how to do it.

I can be reached at dfr1_at_yahoo.com, since my dreamhost webmail is down as well right now ;-)

Be patient and hopefully they will figure it out.

Best Regards,

David

this is ridiculous. it kills productivity and we end up relying on gmail to send emails for work. really, they need to get their act together. FAR TOO frequent and too long of outages.

already started looking for new hosting companies.

March 5th, 2012 at 12:40 pmCraftyLittleHobbit Says:

What this says:

“Umm.. guys? No offense, but you’re getting unlimited everything for less than $10 a month. I’m hosing over TWENTY sites – that’s FIFTY CENTS a month for a website.
If you can’t handle being down for a couple hours a month/year/whatever, and it’s that much lost revenue to you, get an account with rackspace! They’ll be glad to accomodate you, and charge you accordingly.
Do you really think that dreamhost is not on full alert about this right now? Will it really help resolve things if you guys light up their phones and keep techs busy while you complain about something they can’t do anything about?
Oh, and $2 million in that area is hardly a mansion. Check the street view.
Yeesh,
d.i.”

I still don’t have service, that’s almost 4 hours without being able to connect to my e-mail.

Hey we’re offering you a great value in order for us to remain competitive with the rest of the hosting providers, but that value is going to cost you in up-time. Don’t complain.

What sense does that make?

Yikes, this is really not cool. Seems like these outages are passing from unpredictable and reasonable to plain wrong and avoidable. Why isn’t there a proper contingency in place to deal with these sorts of failures? Clients should be compensated by these failures. I don’t currently use my site for business but I’m in the process of setting them up for just that. I hope this is not the service I will receive in the future.

No email service
No panel

hey DI – ummm no im not im paying for MULTIPLE dedicated servers and am down again… 15 hours last month.

@D.I.

A server with a few websites going down, sure I can understand your sentiment.

This is literally hundreds of thousands of websites that went down. This is a massive break which is really bad. And worst of all, this has potential that those thousands of websites may lose information it had. A break like this may mean it can’t recover certain information.

This is insane. Most of our sites are still down.

@D.I.

Thank you for saying that.

i still cannot access email/panel. is this still showing for everyone else?

Warning: mysqli_connect() [function.mysqli-connect]: (HY000/1040): Too many connections in /home/artswebps/pineappleair.artssdc.com/sqli_connect/connect.php on line 30
Can’t connect to database. Error:Too many connections

Is this a DOS attack

Tracing route to panel.dreamhost.com [208.97.187.201]
over a maximum of 30 hops:


9 63 ms 63 ms 63 ms te0-0-0-5.ccr21.lax01.atlas.cogentco.com [154.54
.0.221]
10 63 ms 65 ms 63 ms te3-8.ccr02.lax04.atlas.cogentco.com [154.54.24.
70]
11 62 ms 62 ms 63 ms 38.104.210.174
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.

Well, can’t wait to get kicked out of feeds like googlebase and bing.

Paid feeds to now costing us money but….site down.

Awaiting bad reviews, angry emails, lost sales.

And yes we pay for VPS/ETC.

Thanks dreamhost.

I don’t give a damn about how cheap it is, this is yet another major problem with Dreamhost, are you in business or is this a hobby?

First we had a screw up with our server going down and taking 4 days to do a restore

then we had the Hacking fiasco where you had to change all the passwords

Last week I had 5 sites hacked because you allow inferior security

Now another outage, if this is not fixed soon you are finished, it is just not worth the trouble.

I liek you guys but this is business, if you want to ONLY provide sites to teenagers in their back bedrooms and hobby sites say so on your front page.

March 5th, 2012 at 12:46 pmThe Other Jeremy Says:

all back up and running

@D.I. So, if you paid $1 for a salad bar and got food poisoning, would you then not complain?

still waiting on BLUE HOST switch link…

and everything is still f*cked on the west coast.

hate to say it, im seeking new hosting as well. left 10 years with godaddy to be here, and past few months ive had more downtime than the entire time with godaddy. i dont recommend godaddy in anyway. thinking best bet be to set my own servers up again they never went down. ive very displeased with the responses today, having all your paying customers in the dark as if our monies weren’t of importance. I have 1000+ mb account with dream host on vps, and this is one of 200 accounts we have here as we all left godaddy together. As we will be doing within the hour if we the customers are still left with unanswered situation to explain to our clients.
dreamhost has always been great for the most part, the people there are extremely nice and helpful. I have nothing negative to say towards the staff, im sure they are all working hard on this. But the most important equation of business was left in the dark, thats the people that fund dreamhost to exist.
Please dreamhost i hope you get this resolved so we don’t need to make any move, But i have already backed up all of our stuff daily in the case of things like this happen and will deactivate all of our accounts by 4:30 pm EST.

Does not seem fair that we give this service for which we paid some cantidda each month …. need immediate response is completely stopped my business …

panel working for me now
still no email service

March 5th, 2012 at 12:47 pmThe Other Jeremy Says:

scratch that just cpanel and email up.

C’mon guys…. this is getting beyond what I’d deem “acceptable”. So far this year we’ve had 5 serious outages and it’s only the 5th of March….

no POP mail
no IMAP
no webmail.

website up

Yes this is a DOS attack, dreamhost has a tendency to boot its customers with no explanation and then wonders why so many web-tech savvy people are angry with it.

My sites are all still down and I can’t access the panel. Please hurry up and fix this. It’s very annoying

Hey, is anyone having email issues. we aren’t able to receive email across the board.

I’ve been experimenting w/Linode for approx. 5 months with excellent uptime and performance. You do have to maintain your own system, but I’m seeing better performance and uptime than pretty much all shared and most other VPS solutions.

Good thing dreamhoststatus.com is still up even if all the rest are down. So that we can rant all we want. Thanks Dreamhost, you’re the best!

This has been going on for several hours now. Is there any way to resolve?

Can we get some sort of account credit for all these issues this year.

What happened? Dreamhost used to be so reliable, I have recommended it to so many people, but lately (last six months or so) the outages have just become more and more frequent, and longer lasting.

Tracing route to panel.dreamhost.com [208.97.187.201]
over a maximum of 30 hops:


6 25 ms 27 ms 22 ms 173.167.57.113
7 27 ms 25 ms 26 ms pos-2-6-0-0-cr01.dallas.tx.ibone.comcast.net [68
.86.85.33]
8 56 ms 56 ms 55 ms pos-0-11-0-0-cr01.losangeles.ca.ibone.comcast.ne
t [68.86.87.37]
9 58 ms 57 ms 57 ms as26347.losangeles.ca.ibone.comcast.net [75.149.
228.206]
10 * * 66 ms ip-66-33-201-114.dreamhost.com [66.33.201.114]
11 58 ms 58 ms 62 ms panel.dreamhost.com [208.97.187.201]

Trace complete.

So dreamhost has a 100% uptime satisfaction guarantee.

If your website is down because of them for every hour your down they give you a day of service for free.

http://dreamhost.com/web-hosting/

Scroll to the bottom and you will see a tab called satisfaction guaranteed!

webmail landing page loads, but that’s about it.

email to the client still out

i expect more than adequate compensation will be given not only to supplement the downtime but to also keep me as a customer. There really is no dollar amount you can put on this as everyone is running different sites. I foresee a lot of big e-commerce sites are NOT making sales due to these outages.

TORCH BEARERS UNITE

WE WANT OUR E-MAIL!
WE WANT OUR WEB SITES!
WE WANT OUR DBs!
AND OTHER STUFF!
FIXED!
AND STUFF!

Okay, so I took a deep breath and looked for the possibility of other work to do without a computer, sigh, just glad it wasn’t my system!

I don’t have access to (IMAP, POP, MYSQL, WEBSITE)

It’s a good thing we have another mail relay as a backup, because we are STILL out….

Our email and websites are all down and have been for a while. :(

I hope someone is already fixing this issue…

database domain 100% offline have prevision go back online

How the hell can you cash in on their statisfaction guaranteed BS when you can’t access the panel to submit a support ticket… HAHAHAHHA

Those people saying you are only paying $10 a month for service so suck it are not considering that some of us are paying $100+ a month for dedicated servers from Dreamhost. Currently my VPS sites are up (which I pay ~$75 a month for) but two clients, both with dedicated servers, are down and calling me…..

update: panel down again :(

¿ otra vez ? :s ¿ hasta cuando ? problemas, problemas y mas problemas… cuando sera que pueda recibir el servicio completo, por lo menos por 30 dias seguidos. :(

I hear this is a great host for anyone looking to switch

http://www.nouptime.com/

Two hours and counting, no email or anything.

As my site averages about $150 per sale and this is peak sales time… DH has costed me close to 450+! Moving my sites as this is simply a joke. I could understand a couple minutes but this is a epic FAIL.

Please put our sites back uppppppppppppppppppppppp

Panel is back….

Update from Los Angeles…

WORKING:
FTP, Webmail (Login Screen ONLY. I can’t log in.)

NOT WORKING:
Websites, Mail, Panel, Webmail)

DH 100% Uptime:
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem.

what are people telling their clients i have run out of things to say other than this sux

too baad can’t’ submit a ticket

Thanks for ruining my day…

I still can’t access the Dreamhost panel. Oh my…

The connection to panel.dreamhost.com was interrupted.

March 5th, 2012 at 12:54 pmLaszlo Farkas Says:

Every time I do some intensive work in my websites (once or twice a year), I encounter these outages. Kinda makes me wonder if this is the normal service quality I just don’t notice… I guess this is a push I needed to start looking for hosting elsewhere…

Site, email and panel ALL down and have been for hours.

For less than $10/month hosting, I guess I should expect issues.

No problem, I’ve pretty much stopped developing sites here now anyway.

Same sh*t, different day.

I find it interesting that even DNS is down. This should never happen and is completely avoidable.
If you need a new COO, I’m available… Plenty of experience in blue chip ISP/carrier environments that did not have DNS issues ever!

ERROR:
ERROR: Connection dropped by IMAP server.

oops…panel down again…so close

NO POP MAIL
NO IMAP
NO WEBMAIL
NO DATABASE

DATABASE ERROR ON ALL MY SITES

RUNNING OUT OF PATIENCE

ARRRRRRGGGGGGHHHHHHH!!!!!!!!!!!!!

It’s true – DH will stand behind its 100% uptime guarantee. Just last week I was down for about 3 hours, so DH happily credited my account for $0.50 (yes, 50 cents).

This andrea that keeps posting updates must be a robot.

http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles/

Don’t see her on the profile page… Have a robot take all the flack.

A lot of us long-term clients signed up with DH when it was NOT unlimited and cost more. I’d rather go back to that then crappy unlimited or switching services.

O, AND NO PANEL EITHER

I’m in the Great Lakes region and looking for employment in other areas of the country. NOT HAVING A WEBSITE is not making me a desirable candidate.

I have already found a new hosting provider and will be jumping ships as soon as I can get excess to my DH server.

start spreading the word to your other choice HOST PROVIDERS. Tell them what is going on at DH, let them know they can score a bunch of customers all at once. Lets see what kind of group deals we can get!

still waiting on the BLUE HOST offer…

web panel up
email outgoing, incoming servers still down
squirrelmail at mailboxes webpage comes up but reports 0 messages in inbox. i can’t actually access the inbox either.

Anyone else think that it’s probably not a coincidence that the panel is down, considering that in order to get money back from DH for downtime you submit a ticket, which you obviously can’t do right now! A total scam.

almost 2 hours… unacceptabe

webmail still brokes as jokes ):

This went down just as I was wrapping up my first sales email to a client I’m working on winning. Needless to say this was the last straw. Expect my sites to start migrating off your servers. You’re system has been having way too many issues of late. I’ve just lost all confidence in your services.

EMAIL STILL NOT WORKING

Check out http://www.magnoliasweets.us or et-host.com, great hosting fair pricing.

What is taking so long to get things fixed?? I can handle a little down time, but this is out of control!!!

Just found a cool tool for moving WordPress blogs to a new host (just sayin’)

http://www.23press.com/products/mtb/

@ Sean – BlueHost suck – don’t waste your time.

Dreamhost wants to Touch You. Remember a few weeks ago when they were sending out requests via Facebook to vote for them as best web host or something?

I am definitely frustrated as many of you are – and yes have clients, multiple accounts with DH and rely on them. This IS annoying and we are all hoping the DH gets this up ASAP… I am extremely surprised of the number of people complaining about the $ losing from this being down. If $450+ dollars in an hour is your income (@Joseph), and lets say peak time is only 2 hours (weekday) that’s roughly $18,000 a month. You should host your own with that money or go with a enterprise solution such as rackspace.

Sounds like to me DH customers are a bunch of winers – it sucks we get it…

Some progress … (I couldn’t get through before.)

Tracing route to panel.dreamhost.com [208.97.187.201]
over a maximum of 30 hops:

(…)
5 24 ms 23 ms 24 ms ae2-112.dal33.ip4.tinet.net [216.221.157.177]
6 30 ms 24 ms 27 ms 173.167.57.113
7 25 ms 31 ms 26 ms pos-2-6-0-0-cr01.dallas.tx.ibone.comcast.net [68
.86.85.33]
8 57 ms 56 ms 56 ms pos-0-11-0-0-cr01.losangeles.ca.ibone.comcast.ne
t [68.86.87.37]
9 57 ms 59 ms 56 ms as26347.losangeles.ca.ibone.comcast.net [75.149.
228.206]
10 * 59 ms * ip-66-33-201-114.dreamhost.com [66.33.201.114]
11 58 ms 57 ms 56 ms panel.dreamhost.com [208.97.187.201]

Trace complete.

Error establishing a database connection. Please support.!!!

Come on! I need these email back… Cannot have conversations with my clients.

dreamhost.com is back online but its websites arent.

@ Sigh
by all means please throw out some provider you can get behind and lets get a deal in place with them for all of us.

@Patricia – it’s called a phone

any update????

The solutions to all of your problems: :)
That is, unless you are running a database to get your content from…

Make that: error_reporting(0);

I was thinking of switch to BLUE but I see lots of people saying they suck too … does anyone know of a reliable host with unlimited storage?

@ Sean
Media Temple used to be pretty good, but a bit pricey at $20/month for grid service hosting..

has anyone try hostgator.com?

Still no email. Lost $$. This sucks.

More updates please Andrea.

March 5th, 2012 at 1:06 pmThe Other Jeremy Says:

Yeah i was looking at host gator as well.

u guys can get 99.9% uptime at twsites.com

ive used their service before. good stuff!

We are praying for a graceful recovery to all of DH affected systems. May your servers be surrounded with light, and all domains be restored and alive! We are grateful for the talented and skillful people who are mindful and working for the shift. humbly, liquidbuddha.studios

Still no db connection on our site. Appreciate anything you can do to get us back up – this is a big day for us, just announced a site update to all our users

I lost connection for about 2-3 days of off and on, mostly off, about a month or 2 ago. NOT AGAIN!

anyone use webfaction.com?

All my 10 websites went down!!!!!!! I am bloody losing money!

@Chad… like I said a couple minutes fine but this is going on almost 2 hours. Downtime like this is a pretty epic failure and the lack of updates by DH is alarming especially considering their original statement of “We apologize for the short service disruption.”

Short? LOL

I was fed up with Dreamhost and finally started shopping for new hosts. I used a couple for a week or so during a testing period and if they did not do it for me, I simply quit and got my money back. There is a host that I am pretty happy with so far. They had a couple issues at first which were quickly resolved with a support ticket, but nothing about down-time more just getting my software up and running correctly on their servers.

Anyway, if you want to check up to see how my other site runs on their servers, you can check out http://SpiffyConnect.com and hit me up if you want to know who I am using. I am slowly moving all my 12 sites over to this new host since they have worked out great during the past couple weeks I have been testing with them.

If you are fed up with dreamhost and keep saying “1 more chance” then just get out, its NOT WORTH IT! Go try out some other hosts during their “money back guarantee” period and go with who you like the best, or look around for suggestions.

HostMonster (aka Bluehost) is the way to go.

http://www.hostmonster.com/track/jkrill

If you sign up for hostmonster through that link I get $$$. which is beside the point, they really are incredibly reliable. and cheap.

Fksake, I am sick and tired of this shitty company!

i agree @Chris, rackspace may be a little bit more than I pay now but it beats the downtime and loses. thanks for the link, hadnt thought about someone like them, heading there now

Here’s a brilliant idea, lets piss the dynamic PHP/Python/Ruby based sites off to a different data center to the sites running static HTML and the email servers.

Problems with one service shouldn’t be taking down everything else. If you do anything at all, you MUST run mail, database, and web servers on separate networks.

Now that I’m getting more serious about my website, I am strongly considering moving away from toy host.

@dh-user: That is FUNNY!! Gud wun.

Great going Dreamhost you morons! I got hacked at the same time of your supposed slow-down. I think your royally fked up here! You guys better have backups, because this crap is unbelievable!

still no pop mail, no webmail

“short service disruption” are you kidding

No damn mail, no web service, not even the Panel

Just lost a customer who tried to load my page and called me to say “forget it”

You can’t have sites down and leave it so long, we need updates every 10 minutes

We need you to tell us what is wrong and how long it will take to fix.

Winers, yes well some of us run a business and don’t give me all that kwap about getting a better service, you just make us go

I am off to http://bit.ly/Hostga was given “Remember25″ as a discount code to get 25% off, I have been loyal through so much shite, enough is enough.

STILL DOWN. YOU SUCK.

To those considering Blue Host / Hostmonster (Same Company):

Been there, done that. They use a proprietary CPU throttling technology that make my sites super slow. While they had little to no downtime, the excessively slow response times I experienced were why I left in the first place.

this is absolutely insane.

no e mail no web mail

Time to leave these losers and move on to 1and1.

two words for dreamhost: YOU SUCK!!!

Anyone have experience with FatCow?

http://www.fatcow.com

We,like many others, are also experiencing email problems and your tech support page is not working properly either to even report the situation.

– wf

Ive been chasing hosts for 10 years, they all have problems. Dreamhost is the first that actually acts like they try and fix them, plus the backend and the customer service is pretty good, all for a cheap price.

No i dont work for them or anything, just been there with the “god damn it, im voting with my dollar and moving hosts!” sentiment, and just found different frustrations. If it really means a lot to you to have your sites up, pay someone to set up a dedicated server for you.

my 2 cents as I wait for my databases to get back up.. hope i had the db backup turned on.

Just a point : out of a 30 day month, 7.2 hours downtime = 99% uptime. We are still within that 99% uptime level, and that’s not bad at all considering the price. If this is a DOS attack, then I’d be more angry at the stupid little script kiddies who do this kind of crap just to prove to the world that they do indeed deserve to have their heads kicked in.

A website called Dreamviews got hacked a while back and had their domain transferred to a register in Russia. They said it was all Dreamviews, and they didn’t do shit about it.

I asked Dreamviews about it, and they are keeping it hush hush. They really fucked up guys. They better be working all night to fix this or this will be it for my business and Dreamhost.

My Joomla sites were gradually hacked one by one since yesterday morning. Just minutes ago I got a new Dreamhost security bot message about another hacked site and now I cannot even get to the DH panel! Is someone else experiencing php eval(base64_decode(… hacks?

All Servers Are Down
Our technicians are currently working on permanently destroying the hardware.

LOL

it would be great to get more updates, i need to know whats going on, so i can truthfully answer my clientes

@wren — I tried fatcow. They’re a little lame: constantly trying to upsell you on services you don’t need and their control panel is a custom job that hides a lot of the advanced features several sub menus deep. I came to DH from them. Been happier here…. for the most part.

Website is up. Email is still down.

Still down………

@ tony – do NOT go to one&one – they ripped me off several hundred dollars, and even though I could prove it via their own support emails, the legal cost was not worth the effort. Do NOT use those bast@ards, they are THE worst most unethical greedy bast@rds around. DH are saints next to them.

I am so fed up now going to move everything to hostgator, got a discount code off Facebook of “Remember25″ for 25% off, good enough for me. Been with Dreamhost for over 6 years, I guess their market is for people where uptime does not matter.

Server down.- +2hours

Bluehost is a hot pile of mess, too. Don’t go there from here. don’t even think about it.

Yes we experienced (base64_decode) hacks last week on 3 different sites.

The panel appears to be backup. I can get to it now.

@Otto My Joomla sites started to get hacked a day or so ago too.

Your witty leader (see monthly newslettery) needs to issue some sort of statement regarding the current state of affairs with his company. This is really embarrassing.

Email is up for my domain, send and receive.

Eh.. scratch that.. its dead again..

Jesus Christ.

@Otto, 20+ of my wordpress websites got the eval(base64_decode( hack just minutes before the whole shabang went down.

No doubt it’s connected.

This is a HACK

DREAMHOST WTH, YOU NUT JOBS, GET TO WORK

CLIENTS ARE NOT HAPPY!

@ Otto
the php base64 injections i’ve been experiencing for the past month. first showed itself within WP then started to see it outside of WP. the frequency of re-infection went from weeks to hours up until today.

still getting Connection dropped by IMAP server on webmail and POP isn’t working

I CANNOT BELIEVE THIS! DREAMHOST, YOU SUCK! SO TIRED OF YOUR ISSUES. TIME TO MOVE ON …

shared host is ok, vps is down… funny that this happened when i was about to “rent” a PS and another VPS.

Hey to piss off tony, go ahead and remove that ?affiliate_id=231915 to make sure he gets nothing.

@Otto – I had three of my sites hacked with the base64 junk over the past 10 days. None were on Joomla, all three were in the same FTP space, several other sites of mine (many running WordPress) were all fine. (Never heard anything from DH, bot or otherwise.) One site was running WordPress, another had an installation of Gallery3, two had some lesser-known scripts. I’m not sure how they were compromised, still, but the hacks haven’t come back thus far.

thanks for the update, my vps panel is down, and connection to db is not working

come on — we need email to survive!

Just FYI, Gmail is telling me that it hasn’t been able to get mail from my accounts for almost five hours now. Not a word of a lie. So this isn’t a “short” outage by any means.

1and1 are very very bad. They did an unauthorized charge on my CC. They know it is not worth the effort for someone to go after them for $30-100 so they can get away with it. Avoid them at all cost.

All my e-mails from all my domains are down… was the time that I’m pride to host at DH. I’m sad because with those type of services I need look for another solution after all this years. :(

DREAMHOST GET YOUR SHIT TOGETHER!

PAYING LESS MEANS LESS QUALITY, APPARENTLY.

We expect any compensation, it’s the less thing you can do with those fails…

Despite 1.10pm update above, still no panel, no email no sites.

I am so fed up now going to move everything to hostgator, got a discount code off Facebook of “Remember25″ for 25% off, good enough for me. Been with Dreamhost for over 6 years, I guess their market is for people where uptime does not matter.

My website processes about 7 orders per day on average of 377USD. Wonder how much I will loose…
This is just taking to long.

For those reporting they were hacked.. I was as well.. How are you guys getting your sites cleaned up? I’m still struggling to find where that script is coming from.. I’m not very savvy… This was on a WP install.. happened yesterday.

THIS IS REALLY TAKING LONG ! LETS GO DREAM !!!!!

@ ARGGGG : “PAYING LESS MEANS LESS QUALITY, APPARENTLY.”

Well duh.

You think if you paid more for dedicated server you should get less quality?

Again – duh.

My EMAIL!!!!!! I NEEEEEDS IT NOW!

: (

emails don’t seem to be bouncing when sent to DHosted email addresses

MY PS IS DOWN!!!! PLEASE, DO SOMETHING!!!! I’VE SIGNED WITH YOU GUYS TO RUN AWAY FROM PROBLEMS LIKE THESE… I NEED MY PS/PANEL RUNNING ASAP!!!!!!!! COME ON!!!!!

website just popped up! hope it stays!
cpanel is still down.

@Webdude75 – bad business discussion PAY for rackspace

I expect a free months of service for this downtime, this is not OK.

Not to worry you but my hacked sites were on different accounts, different recently changed secure passwords by usual rules (letters, numbers, punctuation, long enough, random enough): MySQL, ftp, ssh!
I guess we are going to be on TV tonight and on Slashdot in a few minutes.

See how I yelled like that? Well it worked… at least for now I seem to have email again.

But nobody replied to me during that time? How strange.

email works for me now
I hope this will be stable OMG

Whew! Db connection finally back up. 2:45 downtime

Despite 1.10pm update above, still no panel, no email no sites.

I am so fed up now going to move everything to hostgator, got a discount code off Facebook of “Remember25″ for 25% off, good enough for me. Been with Dreamhost for over 6 years, I guess their market is for people where uptime does not matter.

@ Webdude75 : If your sites process 7 x $377 /day on average, which is nearly US$1million a year, then you are a fcuking idiot to host them on a cheap sub$10/month host. You deserve to lose for being such a cheap bast@rd. For $1million/year, you should definitely have dedicated hosting.. hell, it’s a business expense anyway so wtf dude?

Thanks Dreamhost! I still love you!

@CraftyLittleHobbit

I only have one on Dreamhost using cloudflare, but seems to be saving it’s bacon. Only good for static stuff though, I don’t think it will work for dynamic stuff like ecommerce.

I’m back up and running. Just over 2 hours of downtime.

The DreamHost web panel MAY be accessible for MOST of you

MAY = doubtful
MOST = very doubtful ….

I’m back online

fatcow, tocows, godaddy, 1and1…so many more. used them all, switched around for 8 years before finding dreamhost in 2005. the problems i’ve encountered in the last seven years are far fewer than anywhere else.

but even so, this is really frustrating, especially because it’s happening SO much of late. you guys really need to do your testing and transferring at a point that isn’t in the middle of the day. i have a really difficult client baying for my blood.

i remember the dark times at those other hosts and a shiver runs down my spine

wtf, email, web panel, and website down! eta on a fix???

@Sigh
Dedicated servers are down too moron.

THIS SUCKS – I will be leaving dreamhost for good ASAP. WTF is wrong with these people?

March 5th, 2012 at 1:25 pmCraftyLittleHobbit Says:

Back up and running, nice one my little treacles

OK so we’re finally back up….BUT what happened to the emails tht were dent during the time you guys were down??

March 5th, 2012 at 1:25 pmgreenfishfiles Says:

@ DeakPhreak Would that be biznesshosting.net/volico.com? /i got nothing better to do at the moment than robtex for the giggles…

@Kevin

I cleaned out my themes and plugin’s manually, reinstalled wp core, changed all passwords everywhere (DH ssh, ftp / wp accounts / db users and passes ) of course you’ll need to change your config file to the new settings.. also dropped .htaccess into the ADMIN folder with WP.

Thank you! All is working.

@ Hacked Cleanup,

I was installing a plugin for antivirus when it got shutdown. I want to find the hole first, and obviously there is one in Dreamhost. Not sure if we can even patch our own problem.

looks like any email that was sent durning the outrage is vapor

My websites are back up!!! For those that have been hacked go read this http://www.warriorforum.com/main-internet-marketing-discussion-forum/121131-wordpress-mysql-injection.html!!

Also, if you have WordPress don’t forget to always update to the latest version as malware fixes are added.

Good luck

My IMAP mailbox is back, but missing everything that had been received in the hour prior to DH going down.

time to execute emergency GTFO plans…

since i was one of the last to go down, i guess ill be one of the last to get back up…

finally back up… panel moving slow as snot

Our email and website are back. Thanks for fixing things. Although no new emails which makes me wonder what could have happened to any incoming stuff over the past few hours. It’s highly unlikely that we didn’t get a single email during this time. Still, thanks for fixing what was broken.

“My website makes $5,000 per day, and I use a $5.95 DH account for my site.”

“I expect DH to compensate me for lost sales.”

“Since my site has been down for a few hours, I expect a month worth of free hosting.”

Some of you people are absolute idiots.

I guess I’ll have to move ifthispagedoesntloadforacoupleofhoursonamondaytheworldwillend.com to another host.

Panel still not working and it’s showing max connections on FTP, I’m not connected. Any ideas?

email is not getting thru
send yourself a test

@ ARGGGG
You probably suck dick, moron.

We are shopping for new providers also, we really only use the email forwarders. User to use Rackspace, but since they will NOT forward to Gmail, we ended up on Dreamhost. Not thrilled with the lack of spam filtering on the account, but I guess that’s what you get for $5/month!

We’ve tried ImHosted.com, I loved the control panel & ease of management for our 500+ forwarders, but mysteriously after a couple days it quit working and we were forced back to Dreamhost. We still keep it around for DH outages though…

Anyone have any suggestions for providers with great email forwarding options, please post. Must haves: bulk import and spam controls

March 5th, 2012 at 1:32 pmSigh you faggot Says:

Go fist yourself, homo.

March 5th, 2012 at 1:32 pmChelsea_WestCoast Says:

My website is lame, so I honestly don’t care. But email is like the air we breath. I think I’m missing some from 9:30am PST to 1:00pm PST how can I get those?

@Bart, So, if you paid $1 for a salad bar and got food poisoning, would you then not complain?

email back up but no emails received for the last 2 hours when there should be 20+

just got setup at rackspace. gonna miss the nice people at dreamhost, and wish you all well. its a bit more expensive, but business is business

@ Chelsea

Nobody emailed you between the hours of 9:30am and 1:00pm. Sorry.

>> Our email and website are back. Thanks for fixing things. Although no new emails which makes me wonder what could have happened to any incoming stuff over the past few hours. It’s highly unlikely that we didn’t get a single email during this time. Still, thanks for fixing what was broken.

+1

Still no panel btw..

sites working, but panel is not…

PingdomAlert UP:
Trybly (www.trybly.com) is UP again at 03/05/2012 01:18:59PM, after 2h 42m of downtime.

Guys, I’m pretty sure they are not running away from the problems, nor ignoring you, support hasn’t responded to support tickets even at high priority because they may only have 1 or 2 techs on shift and they are doing all they can. I was a customer of bluehost and 1and1, they are over priced and greedy as well as stingy. If you know your stuff or at least are familiar with it, Dreamhost is the best solution for the money. They have a terrific support staff that work their tail off for us, please give them the credit they deserve.

On another note since we’re bouncing I noticed that some sites are up and some are down, connections to mysql is up and down, but ssh is not or appears to not be nor FTP. Email bouncing so obviously we are still having issues. I hope they can reset the CSU/DSU or something and bring the ISP back up to par or something. Some of us process $10k worth of orders a day and all day today we’ve been down. Yea its hurting. Keep us posted. Thanks!

My site’s back up, and my email just came back, but there is a suspicious two-hour hole between received emails running from 2:07 pm Eastern to 4:22 pm Eastern. Wonder if I’ll ever see any of those emails again?

@RR and @Pil

Thank you for sharing your experiences with other hosts.
My site is still not live. However I am able to access FTP with Cyberduck and copying a backup now onto my Mac.

@Jill, DH uses a very aggressive (sometimes overly so) junk mail filter, but you have to opt in.

everything is back on, i lost almost 20 cents in a phonecall due to this, i expect millions in return.

I am just so glad to be moving away from Dreamhost, too bad it takes so much time to move sites and make sure there is not downtime during the move. I really urge you all to start looking around and testing waters elsewhere if you have important or even semi-important money maker sites.

I was one of those people that kept wanting to move away and never took the time to do it until now. Test other hosts under their money back guarantees and make sure your site works on the server, point your domain to their DNS and if you don’t like them, quit, get your money back and try elsewhere until you find one you like!

this is a full on disaster. still no e mail coming through

I got fed up with this bs last outage and started moving my websites and clients.

GUYS AND GALS… GET A RESELLER ACCOOUNT. its cheaper than vps. host your own sites and clients.

click my name for better hosting.

right now i’m, in the process of moving accounts. been busy and again I see why it’s been the right thing to do. SHEESH.

@ Sigh

So, can one infer from your stupid-ass sarcasm that NO-ONE emailed ANY Dreamhost account during the downtime, hmm? Do you think that’ likely, friend, or is it possible you’re talking out of your ass?

SMTP working
Incoming POP or IMAP, not yet…

now we can send, not receiving emails. (pop and webmail)

2+ hours of down time sucks but I sure all providers will have unexpected downtime and DH seems to respond quickly. At least my site was still serving pages that were cached so it wasn’t completely unresponsive. Seems to be coming back on line now.

All of my sites were hacked last week and I was led to believe that it was an issue on my end, and that i should reset passwords. Same eval(base64_decode) issue… had to remove this from all files, delete virus php files, and still I’m getting the auto-regenerating .logs folder in my root directories.

Dreamhost: this clearly isn’t a user issue. Your security has been compromised, and you’re doing nothing to admit fault. I fully intend to not only move hosts but bill you personally for my lost revenue. This is truly a joke. Please handle this professionally.. please, for the love of God.

Site’s up.

Incoming POP working, webmail back up.

for those of you who had your sites hacked, the MOST IMPORTANT thing is to check your usernames, make sure EVERY user has ENHANCED SECURITY, here is how:

http://wiki.dreamhost.com/Enhanced_User_Security Hostgator-code Remember25

Then put a support ticket in asking DH to scan your sites, they will give you a list of what is vulnerable, usually it is a WordPress Theme or Plug-in, it does not even have to be active.

Old version of Joomla also vulnerable, the report will show it all.

@ CP

Yes you caught me – I was being sarcastic. How ever did you figure that one out, genius?

I have not yet figured out how to talk out of my ass. I think I will take some ventriloquism lessons and see where I go from there, but thanks for your interest.

BTW – your name is CP? What is that – child porn?

To give a light to others at the end of this particular tunnel…We’re back 100%. MySQL database access is functioning again for all of our WP sites. Down for about 2 hours this time.

@Sigh Says

Actually, that would not be over a million dollars. It’s $137,228. Just because someone sells something for $377, doesn’t mean the profit that much. It would depend on what their profit margin is. If even half of that is expense, they may only be making $68k. A $150/month is still a lot of money to someone who makes $68k a year. Though, if my business depended on it, I’d probably suck it up and pay it rather than risk losing sales. Especially since it would be a tax deductible expense.

@Dave…Yeah, that’s the same thing…grow up. So, if there is a power outage in your town, do all of the businesses get to sue the electric company for lost sales? No…it’s part of the business, and if your business model does not assume for some problems/downtime, then you are a very stupid business person.

So if “getting contact to your customers RIGHT NOW” is so darn important, just what do you do for the 8 hours of sleep when you cannot contact your customers for something? And if you have employees working around the clock 24/7, then you must be doing well as a company, therefore should be on a server that has redundant backups and high-end tech support. Or better yet, run a server out of your home and run your own T1 line.

Dreamhost going down is a problem. A salad bar giving you food poisoning is willful negligence. I don’t believe you are going to get internet poisoning from DH.

Complaining is one thing…people making outrageous ‘claims/demands’ is another.

how often does dreamhost go down? the last time i was affected by anything was at least a few years ago, i think. i can’t even remember when. seems to me that this sort of thing happens to any host at some point. my experience has been that dreamhost works really hard to resolve any issues in a timely manner. i get no kickbacks and i have no investment in dreamhost other than the money i pay for my plan. i have no plans to jump ship.

All my services are back up. I think DH is not that bad. I can take a little downtime as long as it isn’t too frequent, and no data are lost. People that host mission critical applications shouldn’t use a $15 service! Our company pays $1000 per month for such a service, and have little to no downtime. DH is good for personal web-sites and non-mission critical applications. BTW, I don’t see much data loss in the mySQL database that saves my quiz results, so it is not that bad.

@ Sigh

I think you’re better at talking out of your ass than you think you are. After all, you’ve said nothing but shit so far.

FIRST To Everyone posting here….

I scanned through what you all are saying and all I can say is I’ve been with Dreamhost since 2003, and they are the ONLY hosters I’ve used and tried over and over through the years and RARELY are there ANY TYPES OF OUTAGES!!!! So all I can say is….You are getting a great deal price wise here and the uptime is OUTSTANDING!!!!! Trust me when I say, and after being in the shell business for several years through other providers, Dreamhost has IT ALL AND THEN SOME!!!!!!!! QUIT COMPLAINING and always remember…..RUN BACKUPS because you never know when a DOS attack will occur and believe me THEY DO!!! Nor do you know when they will be needed for restorations! CHILL OUT PEOPLE!!! Otherwise, good luck in finding a provider for your emails, domains and websites that provides OUTSTANDING SERVICE, SECURITY OF YOUR STUFF, INTEGRITY, UPTIME, AND QUICK RESPONSE when you have a question, issue or problem!

To Andrea….Please thank the Dreamhost staff for all they’ve done and what they are doing today for me and those who APPRECIATE them and can wait patiently for resolutions.

Regards,
Dreamhost Customer since 2003

March 5th, 2012 at 1:52 pmChelsea_WestCoast Says:

BAHA! @Sigh –

well, email is not getting thru I’ve sent myself tests during that time frame. And a girl sitting next to me has stuff in her ‘sent’ to my email that hasn’t arrived to me yet.

@ Katie : I believe the dude said average of 7 sales a day at $377 each (at least I read it as each).. that’s how I got near $1m/year – although as you rightly pointed out, that is likely revenue, not net profit.

$377 * 7 = $2,639/day.
$2,639 * 365 = $963,235/year.

Here in NYC my site is still down and I can send but cannot receive emails.

we still can’t recieve emails , just sending.

emails sent to myself within the past 30 minutes are starting to arrive, but anything before that POOOF

OK Panel is down but email & site is up. Panel says

Temporarily Unavailable
The web panel is temporarily unavailable while we perform maintenance on our database.

[Sources: dreamhost]

Which at least means they are doing something

@ Chelsea

Glad someone around here has a sense of humor :)

Usually the backed up emails will trickle through over the following few hours. I hope so anyway, since emails are not being bounced.

Man, there are some real jealous and envious pricks here. Some of us actually do make a lot of money via our Dreamhost websites either by online sales and or driving traffic to our brick & mortar vs the competitions. I believe I’ve even seen some of you chat crap during the last big system outage here. Look, we know you’re just running piddly sites yourselves where making money is a second hand hobby job, but in this economy, when us established big boys lose one sale it can destroy a whole month and in turn send lots of employees packing. Maybe you should think a bit deeper before opening that breathing mouth any further.

March 5th, 2012 at 1:55 pmChelsea_WestCoast Says:

Thanks DreamHost, email coming through.
My little stuff seems fixed. oxox

@Sigh

$963,235 for 365 days. But, this is leap year, so they ought to take that extra $2639 that they will make this year and go get one kick-butt webhost!

@ CP :

Bwahahahaha! Good one, my young ament!

@ Bart : good catch :)

This Dreamhost status shouldn’t even exist. This site seems to show problems occurring daily. Not good.

Hi all! I’m getting emails that were sent at 3:39 EDT just now at 4:54EDT, so it seems like emails from the down-time are slowly coming in. They are not gone forever.

@Sigh Says

I was reading that has he has seven sales a day, the total sale amount being $377. If he’s making 7x$377, I need to know what he’s selling!

Relax have patience, take a little video vacation with Squirrel while you wait: http://www.youtube.com/watch?v=TAmw-f023lI

where are all my emails from the last 4 hours?

The web panel is temporarily unavailable while we perform maintenance on our database.

Seriously. I have no emails from the last 4 hours either.

@ Brain

I eated your emails. Sorry.

Relevant issue here is the hacked PHP code we all got is not our own fault but a system-wide fault. If we are not clear about this, we will be in the same situation in a few hours from now. Any idea how to make DH notice/check/act on this? NOT THIS OUTAGE BUT THIS SECURITY ISSUE?
I thank DH for apparently returning all things to normal. I surely want to be really wrong, as someone rude said before, I hope to be talking out of my ass.

now, receiving messages but delaying .

the email is too slow ….

“Since my site has been down for a few hours, I expect a month worth of free hosting.”
Some of you people are absolute idiots.

Really? I gave the websites Im hosting a free month of hosting because of this. Im not on here crying about the piddly 1 month I pay to Dreamhost. They have been a great provider for me the last several years, but holy crap can you talk about bad timing… just transferred a new client over who had unreliable hosting and I told them we had no hosting lapses in the last 3 years, guess Im a liar now :(

What snippets of code should be be looking for to find these “hacks”? My standard HTML and WP (fully updated) sites *look* clean. I have had to clean out some garbage in the past, from some casino site ad hack on my std website. And the big bad WP hack that went around last year.

Is there new stuff, or old vulnerabilities hitting people who hadn’t cleaned up in the past?

@Bart, Wake up, Low cost doesn’t mean that we should expect bad service – that was the point.

If the cost is too low, then DH should up the price by a few dollars a month and buy more hardware.

Just got an email stating my VPS went over its allocation. Never haoppened before. Wonder if it’s a result of the downtime….

Anyone else have something similar happen?

so what happened to messages sent while the services were down? i havne’t gotten any bounce messages yet on my home email from dreamhost?

Uggh, still down for me inclung the panel. I love Dreamhost, but I will love you more if you fix this.

Our site has been compromised yet again with the exact same symptoms that CJ had posted about: “new /.log directory appears listing a variety of russian urls”. We had JUST cleaned those out this morning and were hit again shortly after the site came up.

I believe the DH security breach on 1/20 is more severe than we were lead to believe.

Hahaha how ironic. I was trying to transfer my SQL databases off so I could change DNS tonight and switch hosts for good…of course it’s down!

Glad I’m almost gone.

Everything was up for a while, now the panel is inaccessible. C’mon guys. Please.

@JTS: that’s exactly what I’m talking about in my previous post at March 5th, 2012 at 2:03 pm

was able to get in to dh panel, most shortcuts dont work

@Ron
check your js files too

Panel loads, I wanted to report the PHP hacks as a support ticket but right now it has been “spinning” the last ten minutes.

noticed one of my mailboxes has been hacked, sending emails to random addresses,
I am in the DH panel but I cant disable it or change password as of now.

Are you people aware that panel is still down?

Temporarily Unavailable

The web panel is temporarily unavailable while we perform maintenance on our database.

Temporarily Unavailable

4:20pm central:

https://panel.dreamhost.com/index.cgi?tree=billing.accounts&amp;

Temporarily Unavailable

The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

@Sean
did. thx. seems clean. but i’d like to a global search from time-to-time to keep an eye on things, and i don’t remember what the “signature snippets” of code were from the last infections. i remember they were pretty obvious though.

mine is up , but most links dont work

I cannot get into cpanel to contact support. My site is still not up. When do you expect it back? Can you check it out?

Dan , hit refresh and it may work

All my sites that use wp are down – thought they said they had them up and working around 11:00 am… can’t even get into panel to tell them i’m still down :(

You give a link to contact support about the outage… and the link is down.

What a complete joke.

@Dave…glad to know that in your almighty wisdom that YOU get to define what is ‘bad service’. If you’re that disappointed with DH, then leave. I doubt they’ll miss your $5.95/month. I, for one, am happy for what I get verses what I pay. If I’m not, I will simply move elsewhere and pay more. Easy as easy can get…

Do I like this downtime? No. Did I mention that is was stupid for people to COMPLAIN about this problem? No. I made light that people are asking for some very unreasonable things, and THAT is being stupid.

Hmmm…bad service…they are working on the problem, and trying to do what they can to keep us updated as to where they are in the process of fixing the problem. I hardly call that ‘bad’. Better than what you’ll get at some other places that cost you more and promise you more. I forgot, in your world, nothing EVER goes wrong that you have to fix…must be nice…

To the people equating the russian /.log hack with this downtime, that is not the case. The two things are totally unrelated. Yes, there are clearly compromised accounts on some DH servers (Columba, definitely), but if the /.log hack affected one of your sites, it is almost certainly because you have directories with 0777 permissions. Look in your 0777 dirs and I bet you’ll find files called r.php or rr.php – that’s the backdoor the hacker used to get into your web directory.

Hey hey… goodbye… na na na na, na na na na, HEY HEY, GOODBYE!!!

Can’t contact support, can’t use control panal, can’t do anything. Considering we signed up TODAY, this isn’t a very good first impression… Would love to be able to register a domain name before the day is out…! I really hope this is a one off event otherwise I fear I might have just wasted my money.

I am still having issues with my site and unable to access the contact page for you guys

I STILL HAVE NO WEB SITE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Charlie – no, there’s been *serious* downtime lately. They’ve promised to improve but no progress so far…

Luckily they have a money back guarantee for something like 90 days.

TRYING TO SUBMIT A SUPPORT TICKET BUT CAN’T GET INTO THE PANEL. I LOST MY SITE MOST OF LAST WEEK SINCE I WAS ON STAMPEDERS, NOW I DON’T HAVE A SITE YET AGAIN, YOU’RE DESTROYING BUSINESSES DREAMHOST AND YOU DON’T GIVE A DAMN!

No cPanel or support access:

https://panel.dreamhost.com/

Temporarily Unavailable

The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

@Paul: “This andrea that keeps posting updates must be a robot.

“http://dreamhost.com/about-us/the-dreamhost-team/employee-profiles/

“Don’t see her on the profile page… Have a robot take all the flack.”

Her real name is “AndrAIa.”

(Obscure?)

My sites are still down. I think my issue has been forgotten in all of the chaos. I’m sure that my site being infected with some trojan/phishing software was no coincidence but I need to be back so please have your abuse team look into this matter ASAP! It’s been 16 hours and all of my sites/email/sql everything is still down.

You’ve disabled my account so I can’t even access my databases to replicate and move to another sever. This is absurd. 10 years of web design down the drains?!!

Thanks!

My site is down, as well as yours!

As it says in your policy, I want my months refund back!

Still can’t access my email folder. Error message at 2:32 PST (15 min after you say its working)
Temporarily Unavailable
The web panel is temporarily unavailable while we perform maintenance on our database.

I HAVE NO PANEL AND NO WEB SITE AND WHEN I EMAIL SUPPORT I DON’T GET RESPONSES!!!!!!!!!!!!!!!!!!!!

Temporarily Unavailable

The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

We have access to some cpanel pages but not others, including the one to file a support ticket request. Please keep working to resolve this.

OK it says things are fixed but our website still down at this point. And I can’t get the contact support link to work as it says it is down too. :-(

I love losing money, so why have not you?

im tired with this problems… its time to looking for another host.

Second time in a week we’ve been paralyzed by 500 errors when no changes have been made to the site. Status updates tell me that sites should be up and the panel on line. (No indication of problems *last* time on the status page…)

Site unusable. Can’t even log an incident, as the web panel is unavailable.

We are not a high-traffic site, and I’m beginning to regret paying that year in advance.

March 5th, 2012 at 2:43 pmiamontheinet Says:

Web Panel is *not* available. I get the following message:

Temporarily Unavailable
The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

Panel is still down, can’t contact support without panel.

Yes the panel is still down!

been with dh for 10 years and this is really a bad one! As a small business owner I believe it’s time to invest in something better. I guess you get what you pay for and maybe not even that.

-concerned-

Ps6227 down.
I’ve been 6 years working with Dreamhost and have decided that an era has ended. I lost 5 clients, I have a web of 40 families that depend on out of service for more than 150 hours. It is a wreck and I do not happen again.

Yep panel loaded login screen but won’t log in yet.

It seems that there is a fire in one of the DH datacenters…

@Jim, joke? How you know this?

My panel and my sites are still down. I can’t even submit a support ticket.

@chad because a lot of sites are still down and it seems support ran away telling people everything is fixed… but it is not :-)

Panel still down. :\

panel is still unaccessible and some email is not coming through imap. it’s 4:48 CST

my site has been randomly going up and down and I tried to contact support through the panel and it is still not working.

Thanks.

When can I expect a credit to my account for this downtime?

Everything was fine mostly throughout this outage. Now our site is down and panel is giving the error:
Temporarily Unavailable

The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

My sites are still down as is the panel.

I like the 2:45 update from – Dan w

“Please contact support with details, if you’re still having issues with your site services. – dan w”

Contacting support is via the panel. The panel is down, therefore one cannot complain. Maybe there should be an alternate method to contact support on the dreamhoststatus.com page..

Well…it was up…and not’s it down…again…and “checking status” just spins and spins while “fetching” results….hmmm

@Matt

I’m fairly sure I saw their employees setting up a sled run on a slope paved with money, so I’m guessing that they’re not having financial problems. Networking problems possibly, financial problems no.

Our websites are still down.

“We’ve done extensive testing and have determined that affected customer services (site, email, mysql, etc) are functional again.”

WRONG! All of my sites are down and the Panel is not responding so I can’t report it.

Wow…up again…this is like a ride at Universal Studios (the Florida experience)

Webpanel is DOWN DOWN DOWN…

Can’t contact support IF I can’t get INTO the Webpanel.

Plz Help!

All my sites are working perfectly.
The panel is still out with a message “Database Maintenance”.
Now, place the link to contact and that it leads to the panel that is down for maintenance was joke huh?

major outages every 2 months now – or what!????????????

It’s kind of hard to contact support if the link takes me to a “maintenance” web page. :-P

Still down, and can’t use contact support to contact support…

I’m buying from GoDaddy now.

Stop all of your complaining – you get what you pay for. I have a website with a hosting company that takes 3 days just to get back to me and all of their help topic links are broken… so the fact that we have updates and all of this conversation, I would say that they are not doing bad for the price.

I’m the TI guy in a company that use your webmail, and one of the employes are still waiting the webmail service to come back to its normal, because she have to pay a billing and in here (GMT-3) are almost 8pm, so the banks will close.

She alreading receiving some e-mails(the ones that has been sent to her as a test a few minutes ago) but she needs and e-mail that has been sent to her a few hours ago, and she didn’t receive it yet!

Please solve that soon!!!

yea.. go to godaddy.. hope you enjoy their dashboard… Crazy. and lots of upsale loops..

Our site’s .ssh/authorized_keys file was edited this morning before the outage. Is this something to be concerned about (e.g., a hack), or is it an artifact of some automated DH process?

GoDaddy is absolute garbage. And their crimes against Good UI are unforgivable.

im getting this in my panel

Temporarily Unavailable
The web panel is temporarily unavailable while we perform maintenance on our database.

[Sources: dreamhost]

This is ridiculous. The panel has been down ALL DAY. Dreamhost just IS NOT reliable. This level of incompetence is unacceptable.

Update Mar 5th, 2:45pm PST: We’ve done extensive testing and have determined that affected customer services (site, email, mysql, etc) are functional again. <<< I beg to differ, I have several domains who's email is *still NOT working* and I cannot submit a ticket since "The web panel is temporarily unavailable while we perform maintenance on our database." ffs

Almost every PHP file on my account has had the base64_decode code injected into it.

GoDaddy’s pretty bad, no doubt about that. But it’s up and running right now – which is when I need it.

Site still down.
Panel still not working, so difficult to contact support via that posted method.

Expect better

E-mail still not working…
no access to support or my web panel.

It’s definitely not fixed yet. I can’t access the web panel to contact support as you suggest. My website, roryparle.com, is still unusably slow.

March 5th, 2012 at 3:15 pmAlex Ramos de Oliveira Says:

Panel still not working….

Finally got to the support ticket screen, now it’s been stuck on the spinner while ‘checking status’ :(

No access to panel and cannot connect to update web pages

I have contacted Dreamhost support via support@dreamhost.com, but I don’t have any response yet.

James: “Almost every PHP file on my account has had the base64_decode code injected into it.”
Same happened to me. There’s some form of a security breach that occurred that DH isn’t disclosing.

To the other James above – ‘Almost every PHP file on my account has had the base64_decode code injected into it.’
That would be due to vulnerabilities in the code/app you’re using, not due to this current Connectivity Issue…

Yeah… I would LOVE to contact support. But as the panel is down, that’s not happening.

@Ted
DH isn’t responsible for making sure that the user’s code or apps are secure, that’s up to the ‘admin’ or code writer, whether it’s something like phpBB or self written code, DH aren’t the ones who wrote it

Panel still down.

I also have had the base64_decode injected into my site. So unimpressed.Fix this dreamhost!!!

James: “Almost every PHP file on my account has had the base64_decode code injected into it.”

Yeah – same here too – has happened three times in the last month – apparently it’s a WordPress hack/exploit, except I’m not running WordPress, but as I understand it, the hack when it occurs runs globally on a server.

I’ve had to move my most critical domain to a VPS to hopefully ensure it stops occurring.

I’ve had the PHP base64 code injection too. Running the latest version of wordpress. Seems that it could be down to the host and not the code. Look up godaddy wordpress hack for details of a previous time when a hosting company admitted it.

James,

If you want to find all the changed. in particular one you know you have not touched, just ssh to your box and run this.

find . -type f -mtime -3 | grep -v “/Maildir/” | grep -v “/logs/”

You should see a list of all the files that have changed in /home/yourusername

panel still down…

All of my sites got hit by the base64_decode injection too. 6065 PHP files are infected across my WordPress and non-Wordpress sites.

I been using ezprovider for years now, never had an outage like this with them.

Mike its time to rm -rf * and restore from backup.

I’m in France and https://panel.dreamhost.com gives me:

Temporarily Unavailable
The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: dreamhost]

however, if I use a proxy to “look like” I am in the USA, then https://panel.dreamhost.com is working OK.

Is it a known issue ?

(Please email me if you have an answer – thanks)

I’d love to contact customer service… but the page cannot be displayed… I really regret not changing my mind before the trial period expired…………so much for opting to a hosting company because I didn’t want to risk my home server being down…

“Almost every PHP file on my account has had the base64_decode code injected into it.” We had the same thing on one of our accounts. DH, what’s the deal?

FYI.. I have been waiting on this to get resolved all day and almost made a mistake to suggest DH for company-wide use. This outage and all the comments came in good timing.

Contact support, eh?
Fat chance.
Support link isn’t working.
Neither is my webmail.

Sites appear to be up, but I cannot get into the Control Panel. Accordingly, the contact support link is not working either.

“IMPORTANT INFORMATION: We are writing to let you know that there may have been illegal and unauthorized access to some of your passwords at DreamHost today. Our security systems detected the potential breach this morning and we immediately took the defensive precaution of expiring and resetting all FTP/shell access passwords for all DreamHost customers and their users. There are three different types of passwords at DreamHost: a web panel password (for logging into the panel), email passwords, and FTP/shell access passwords. Only the FTP/shell access passwords appear to have been compromised by the illegal access. Web panel passwords, email passwords and billing information for DreamHost customers were not affected or accessed.”

I received the above before I even coded or hosted anything with DH, I just had the service and delayed use of it…this wasn’t phishing, came from DH. Seriously, how much of it does one blame on the end-user? It’s clear that “Jame’s code” isn’t the only thing with vulnerabilities.

callepontebella – It is not DH responsibility to keep your site’s applications secure and up-to-date. If you are running the latest versions code you would have not gotten pwnd. The base64 code injected is a well known hack..

email is back but panel is still down.

Panel is down

@Chronix: Joomla 2.5.1, no plugins/addons, installed last week, just got hacked with the base64 code.

For what this post say is most of the issue resolved which is a lie. How I’m going to contact support if the panel still is down ? I have 2 servers down. My customers are waiting. Do you have an ETA or something ??

I have BPS security, WordPress File Monitor and scan my site daily with Sucuri and I still got hit with the base64 malware. It is definitely not the end-users fault.

email is sometimes back. Panel is still down, mySQL access incredibly slow/intermittent. Can’t reach DreamHost support to start a ticket.
Have had terrible access times and delays for the last 4 days. Not getting answers to support tickets already entered.
What happened to Concrete5 installs?

I’m hoping this is viewed by the staff. I can’t get to a contact page, from any direction, to report that we are still having issues — panel and some webmail. Our sites are slow but seem to be up.

All services still down with the exception of outgoing mail. all my personal sites and client sites are down. Ironically my main site is for spitting venom on things like this, cut off at the source! I have lost a full night of final stage development for a client on a very tight deadline, and this is a full night lost… not good! and actions speak louder than words, dont say its fixed unless its fixed!

is this the same message everyone else is getting?

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@lupanast.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

more BS from dreamhost — what don’t they hire some real engineers!@!!!

THIS IS BULLSHIT!!!!!!!

NEVER EVER EVER SPENDING A SINGLE PENNY MORE ON THIS COMPANY THAT HAS NO SUPPORT AND SHITTY OUTBOUND CUSTOMER COMMUNICATION!!!!
And stop with the apologies, just fixe your datacenter!!

Otto – you may want to look at this forum post http://forum.joomla.org/viewtopic.php?p=2692360

Pretty much, spittingteeth.com, except the last line is a 404, not a 500.

My site is running again (barely). Panel doesn’t give the “maintenance” message any longer, just a browser error:

The connection was reset

The connection to the server was reset while the page was loading.
* The site could be temporarily unavailable or too busy. Try again in a few
moments.
* If you are unable to load any pages, check your computer’s network
connection.
* If your computer or network is protected by a firewall or proxy, make sure
that Iceweasel is permitted to access the Web.

Can DH at least give us an update on the situation? I need to make changes to a client’s site, but the FTP password was reset and the control panel is down…

I’m stepping out of Dreamhost ASAP. A server in my house would be more reliable

Yea Panel Down – Nothing to do but wait.

DreamHost Team has always been helpfull and a great support team. However like you this hacked stuff is driving me nuts.

I am working on cleaning our stuff out but this is just ridiculous.

I think DH Should clean everything up for us… This issues obviously came from them.. I dont understand how or why but DH in total appears to have been hit hard.

Although your update says the problem is resolved, I still can’t access my panel, and I’m more than a little frustrated that this is the only way to contact you. Angry, actually. Very angry.

thanks Charlie, just wanted to make sure i wasnt getting something entirely different off the back of the downtime…

OMG! Can you people keep your services online long enough for me to finish transferring all my stuff away?! DIAF dreamhost.

IF YOU ARE GETTING BASE64 HACKS:
Out of curiosity, which server are you on?

Y sent an email to support@dreamhost.com and some minutes later I received an email back with the subject “Your support message was NOT received!”.

cant remember what server i am on off the top of my head, and of course i couldn’t double check as it stands….

Hey guys check out my website http://www.trillerthanmost.com , that is if your browser doesn’t automatically redirect you to a malware warning or website that doesn’t resolve at all. this happened once previous last week, I cleaned up every file manually, and now back to the same thing. This absolutely needs to be fixed. If anyone can help me with this issue, please get in touch!

March 5th, 2012 at 4:09 pmDreamhostClient Says:

Well my site is working … but I can’t access my panel … Ironically, Dreamhost want to get paid and the link to the panel is what’s not working
This is really ridiculous … anyone got any recommendations? … I think I’m changing providers!!!!!

@CJ: I’m on thunder.dreamhost.com. Half a dozen sites base64’ed.

Dreamhost Panel still down for me, in San Francisco Mission District with a Comcast internet connection.
Come on Dreamhost, get your act together.

Still down for me too… waiting on more updates

One of my WordPress sites was BASE64-hacked as well. I’m on the server “herediano.” This isn’t the first time it was hacked. Now I’d like to restore a backup and recreate the site on a fresh location, but alas, I can’t access the Dreamhost panel since this morning due to this outage (and still can’t). Seriously, what the hell?

This is the least of my problems, I noticed today that whenever i went to my site the page would jump to one of those scam you got a virus on your computer sites, at first I thought this must be something on my computer doing this, so I checked on a few other computers and exactly the same, so i log into my dreamhost server and every PHP file had been stuffed with this code at exactly 4.48am March 3, I’ve been uploading back ups all evening and my sites back to normal. what could of done this?

4:00pm PST and my sites are still down! I run a very small business (nights / weekends from home). This outage is frustrating to me….must be unbearable for people whose only income depends on their website. Awful.

Panel is still down for me. Located in Ottawa, Canada at 7:11 EST (4:11 PST)

Here’s a little cleanup script I wrote – there are probably others out there, but I just threw this together and it worked great. My sites are all clean now. Don’t blame me if it breaks something for you though – making a solid backup first is YOUR responsibility!!

Hopefully the forum won’t break the formatting here.

sed -i.base64inject.bak ‘s///’ `grep -rlI –include=*.php NvZGVkX2NvbnRlbnQuZ21sXzc3NygpOyAgfSAgfSAgb2Jfc3RhcnQoJ21yb2JoJyk7ICB9ICB9 *`

That’s supposed to be all on one line. Just paste it into your Bash shell and let it work. In a nutshell, it will:
1. Recursively locate all infected .php files under the current directory
2. Make backup copies (*.php becomes *.php.base64inject.bak)
3. Clean them!

Once you’re confident the cleanup was successful, you can wipe out those backup copies with this one:

find . -name ‘*.php.base64inject.bak’ -exec rm {} \;

Hope that helps somebody.
Mike

Sites have been up-and-down ….. Can’t navigate to control panel to post a ticket. Because https://panel.dreamhost.com/ is returning a “101” error.

Yeah of course the forum broke it. Here are Pastebin links:

Cleanup script: http://pastebin.com/KA4Z5Amq
Remove backup files: http://pastebin.com/D7ZhCsDH

Still cannot access panel, experiencing extreme site slowness on simple scripts, and to top it off I cannot access the support page. I have sent a tweet to @DreamHost @DreamHostCare and @dreamhoststatus. These problems seem to happen every time I have a deadline.

oh come on i am gonna kill myself here

i would love to contact the support, but sadly, i cannot access the page because it still isn’t fixed. this is pretty much 2 days straight i haven’t been able to work on my site

Confirming….. The panel is out of service in Latin America at 7:17pm.

Wow. I have been needing to change my page address all day this is no good….saying server not responding again. Not good when other companies are relying on me….in Chicago 1t 615 pm

Site is down (STILL) and I can’t even access the site to contact support. It’s like a sick joke to give me a link to report problems that won’t even load.

its not working.. all my WP sites are down… and i can’t login to panel to tell you!!!

So, the site is slow, but I can’t get to anything via http://FTP... What the hell do I do???

Yeah….site is still down. So much for everything being restored. And I wish I could contact customer support…but I can’t.

Just tried to access my server’s control panel to make some user changes. Connection is being reset. SSH is not working at the moment either: ssh: connect to host mydomain.com port 22: Connection refused . 07:22 AM Bangkok Time. Sites on the server and email do seem to be working.

I always heard good things about dreamhost and was looking for a new home for about 10 domains of one of my clients, just signed up to transfer first domain, noticed payment system was slow to process, and now this. I guess I’ll look elsewhere.

“Everything” is not “up and running” as of 4:15PM Pacific Time. Still getting 500 Errors across the board. I understand downtime happens, but how about more accurate reporting..?

Cannot access my site via FTP to do anything to it or check if I’ve been hacked or anything… Help!!!???!!!

Everything is not up and running. My e-mail server is still not working.

Would be fantastic if Dreamhost took over https://getsatisfaction.com/dreamhost and used it instead of their baked-in support.

@ Mike – That script you wrote… My base 64 code looks a bit diff.
What is see is

I dont suppose that same code would work for me in this situation – Do you have something that might work similar.. While I can design like crazy I am useless at coding things like this.

March 5th, 2012 at 4:40 pmRoger Morris Says:

Anyone needing help solving base64 hacks who runs WordPress might want to contact https://www.andrewvilleneuve.com/ – he’s a genius. Well worth it if you want your site back up and running in a timely fashion.

March 5th, 2012 at 4:41 pmRoger Morris Says:

sorry, malformed link. https://www.andrewvilleneuve.com/

Well since the code got messed up here – I will attempt to make it look diff – / * * / eval ( base64_decode ( ” aWYoZnY etc…. ” ) ) ; ? > Hope it shows this time.

ALL of my sites are still down!!!

Luckily for me the changes I was half finished making didn’t leave my site in a screwed up state. I’m guessing there were others who weren’t as lucky as me…

Our site remains down.

support@dreamhost.com

That might get their attention if their email is working…

I seem to be in the market for a new vps hosting company.
– don’t tell me GoDaddy
Anyplace else that supports decent mysql and ssl? Pretty much anyone…
but looking for recommendations at this point.

Petition Dreamhost to use an external service such as GetSatisfaction for their Support Site —> http://act.ly/5na

Same with me. Thousand of php files ruined. Sites all down .. this is really bad

HOW THE F*K ARE WE SUPPOSED TO ‘CONTACT SUPPORT’??

Ah, I’m on _sarajevo_
Was thinking it might be a localized thing.

NO ANDREA, everything is NOT RUNNING. I have 2 VMs down!!!!

I run a wordpress network on Site5 hosting. So far so good and their support team has been great. I still unfortunately have a few sites on Dreamhost that have been down or only working every once in awhile for most of the day. This, after having all emails go down a few months ago is making me want to move everything over.

That being said…I STILL HAVE SITES DOWN. wtf?

OK now my posts are getting held for moderation… for being too helpful? :) Hopefully this one will make it through.

Cleanup script: http://pastebin.com/KA4Z5Amq
Remove backup files: http://pastebin.com/D7ZhCsDH

The base64 code you see in my script is just a snippet of the real thing. And Sparker, I don’t see that “aWYoZnY” bit in my backup files. I guess you’re right – different sites may have been hit with different variations of the code. You can try replacing the base64 snippets in my script with “aWYoZnY”.

This is a LONG time to be unable to access our panel and make security changes to our FTP directories.

ps27196.dreamhost.com VM is down. NO SHELL.

@JM – If Mike is correct above he may have found a way to quickly clean them.

We will see if I can get a reply since it appears that alot of the scripts are different.

Also for those of you that dont remember or dont know – Godaddy got nailed in 2010 with this same crap.

Website just came up.

Also have had the base64_decode injected into my site on Thursday. Dreamhost was helpful in helping over the weekend. Today of course is just nuts. No email, all connections to server LOST…

This issue really needs to be fixed ASAP. This is extremely unacceptable for this many people’s websites. This is extremely detrimental to many businesses that rely on their company websites. PLEASE GET THIS FIXED ASAP. How would DreamHost feel if their website went down and NO ONE WOULD BE ABLE TO TAKE ADVANTAGE ON YOUR SERVICES. Well this is what happening to OUR websites. Do something and do something FAST.

No access to panel, no way to send a message to support since it tries to route me through the panel. Any updates?

Dreamhost is obviously not ready for the enterprise. I will be pulling my sites 190 websites from here ASAP

Anyone fighting the base64_decode(”aWYoZnY… hack on their WordPress site, I found a tool to clean it up. Pretty sure it was a plugin that allowed this, but still haven’t figured out which one.

Make a backup of your site (just in case something (else) goes wrong) then run this script on it.
http://www.php-beginners.com/solve-wordpress-malware-script-attack-fix.html

Base 64 f..d dreamhost all over the world they dont have any chance to get up unles they clean all the infected files and it looks impossbile.

My site is still down, my WordPress admin is down, and I cannot even access the logon for the Dreamhost panel.

I can’t believe this problem hasn’t been fixed yet.

I’ve been trying to reply but my posts are held for moderation (boo!!)

I’ve put the script on my website. Im afraid if I put a URL here I’ll get moderated again, so click my name above, or to to www dot michaelkerley dot net.

The only service still down/intermittent is the DreamHost web panel. If you’re having other problems, it’s unrelated. You may contact us with the details through http://dreamhost.com/contact/ . Please be as specific as you can, as we can’t reference your account and case without that.

Panel is down again for me and WP site is responding very slowing or throwing 500 errors – things to not appear all “ok” yet

March 5th, 2012 at 5:29 pmAnother unhappy dreamhost customer :( Says:

It 8:30 EST, and while my sites are visually up, I can’t access any of my emails, and even the one that is hosted by google can not send outgoing mail. This is a huge inconvenience. I can understand a service interruption for a few hours, but an entire day? This is the first time Dreamhost has truly pissed me off.

If by unrelated, Andrea means there are still thousands of websites on their servers with based64 trojan viruses that were injected via their bad security …..

by the way, if your site is experiencing a 500 internal server error, your WP page has likely been hacked. this happened to me starting last week, and is affecting several users.

March 5th, 2012 at 5:36 pmBruce Thompson Says:

My domains are still acting up, either exceptionally slow, or just plain not responding. Can we get an ETA on everything being back to normal??

Is there any discussion about the based64 trojan any where? I’ve received an automated report identifying nearly 50,000 php files that have been infected across about 6 – 8 domains / sites. Is this a problem Dreamhost is resolving, or something we’re having to address site by site? At this end, I’ve rebuilt two major projects already.

@andrea: Tomorrow dreamhost will be a trending topic on Twitter, just because time is on your side. Most people are back home by now. How can be unrelated that we are all experiencing base64 injections? From simple one page PHP custom scripts to WP/Joomla/C5!

@Andrew Garton: See Paul Smith comment way up. I repost here the URL he posted:
http://discussion.dreamhost.com/thread-134262.html
First post is on Feb. 17 I think. And no, DH is not doing anything. It is our problem for being so dumb and lazy with our outdated Joomlas/Wordpresses and else.

Just checked my server. Somehow there is .htaccess files on every domain with malware links. Trying to get Google to un blacklist me. Just FYI for people to do some checking on their servers for .htaccess injections before google crawls and blocks you. Seems like there is some security holes on our servers. qimulti is kicking dreamhosts butt.

Panel just started working for me. Now I can finally finish setting up my backup cron jobs…

Wow I just signed up and paid for a year’s worth of hosting last night.. I hope I can get a refund because this isn’t a good first impression

March 5th, 2012 at 6:03 pmWe are not hacked Says:

*Your* site got hacked because you were lazy and left old installs and plug-ins laying around.

I have several sites on DH and the only *one* that got infected with the PHP base64 encodes was one with an outdated WordPress install.

@We are not hacked: I hope that stays true in your case. I thought the same as you last night: some of my client sites got hacked, but not me (they manage their own accounts). On my account I have only one basic Joomla 2.5.1 install with no plugins for test purposes. Now it is injected.

@We are not hacked: no wordpress no joomla just html with some basic php. I was just warning people to go check, because every one of my sites was infected. Some of which were just html, css. Just warning you will want to check for .htaccess injections. Don’t just check wordpress sites.

i wish DH was honest with us. We should know what the hell happened. Just found 3000 spam files on my web site.

IF DH IS REMOVING THE MALWARE PLEASE TELL US – – SO WE KNOW *NOT* TO REBUILT THINGS !!!!!!!!!!!!!!!!!!

Every PHP file in my account has base64_decode injected, even for sites that haven’t been online in years…

or if anyone can offer a fix for bloggers who are hosted by other users and don’t have FTP access at the moment, that would be insanely great.

For those of you just now checking in – Please see the latest post from Mike above. His fix and the one from Nick seem to be cleaning things out quite well I am double checking everything now but by tailing my log file it all looks to be working correctly.

Most of my sites are rather large so it is taking a while to complete.

I do suggest running the command line versions of either one as the DH server will time out.

Support this petition to Dreamhost to use an external service such as GetSatisfaction for their Support Site —> http://act.ly/5na

Are people still finding these infected PHP files on their servers? I found mine on February 17, and the files were injected on February 14 (some kind of Valentine’s Day prank?).

I’m curious if people that are finding this problem, have any files on their server that they know that they themselves wouldn’t have changed (or changed by any known processes), and what the last modified date of these files are.

Like I said, when I found mine, I found them on February 17 (and fixed), and the injection happened on February 14.

Mine just got infected last Wednesday – When I asked DH about it they said it was something on my end… But now I am seeing that it is actually their servers So we as a community just simply need to come together and resolve this.
The same thing happened to Godaddy – Blue Host – Network Solutions and several other hosting companies through out the years. So we just need to make this work OUR way – Obviously we are solo in this situation. While DH is usually a good host this is just something above them.. We as True web admins need to stop whining about our host and deal with this BS ourselves.

My control panel is finally back up. The outage lasted seven hours by my calculation. Not a good day, for sure, but I’m ready to move on, assuming everything is back to normal now.

@Sparker

Couldn’t agree more. At this point, people should stop worrying about whether or not DH lied, covered something up, etc, and simply go fix the problem. I had hundreds of files infected and it was cleaned up in less than 1 hour.

Fix the problem, then find ‘fault’, then decide if action is necessary (switching hosts, etc).

My WP sites are still giving 500 errors for non-cached pages and admin panel. I can still FTP into the sites and no sign of any hacks – just the unresponsive mysql servers.

Again id like to state that just last week I got hacked via .htaccess and was given a 500 internal server error. We cleaned all the files. I installed BPS Security & Word Press File Monitor. Updated all of my plugins. Deleted unused plugins. Deleted unused themes.

Today, with all this other nonsense I got hacked badly again, redirecting me to some page that blacklists me on google.

This is NOT your fault WordPress user. It really isn’t.

The clean up script i posted isn’t quick, but it appears to be effective…

I’m not really sure how the infection started but all php files are infected, even those in sites that haven’t had a functioning domain name in over 2 years…

Nick I got to say man — Your script worked as well as Mike’s – it took 61 minutes to clean a Concrete site that was fairly small but contained a ton of php files…

We are clean and clear at the moment – I am not sure how long it will last but this is the part where DH comes in and resolves any security risks they may have a hosting company.

We as a community are kicking butts in this resolve. We will all probably get an email tomorrow with a request to change our passwords again… YUCK…. But regardless we as a cumminuty of Web admins will take charge and work this out ourselves…

I personally am sitting here with some of my employees kicking the crap out of this issue.

I want to thank the guys that posted fixes for these things…. You guys are awesome and the rest of the people here should commend you for the effort you put in.

Regards,
Sparker

Nick and Mike’s fixes seem to be manna from heaven – could anyone spare a few minutes to explain to a newbie like me how to employ them – as if I’d never seen anything like this before? I am extremely new to a handful of sites I inherited and am running for a set of nonprofits – our “team” is me, and I’d really appreciate a handhold how to dig out from this. My wordpress sites were up to date, but I don’t know how to see if they were injected, and my joomla site is absolute toast beyond the home page.

I know it’s probably tempting to beat up the newb, but I’d so appreciate the help if anyone out there has the patience. Thanks!

Oh, please… still problems… :-( lately dreamhost is becoming nightmarehost, sorry to say :-( so much don’t want to say goodbye to you guys, but please… fix it.

March 5th, 2012 at 8:10 pmJoseph Croasdaile Says:

site was up for awhile… now all back down

All my sites are still down. Email has been unreliable for over a week. Worst part is I just went live with product for sale last night. Thanks DH, way to make for a successful business. I can’t do business this way. If I don’t have my sites working in the morning with an answer from DH on how they are going to make sure this kind of BS doesn’t keep happening, I will be migrating to Bluehost or somewhere else.

timing of this cluster = not good!

Our WP site databases are on snap:tuckey and its definitely having problems still – sites are down with 500 errors and phpMyAdmin is barely responding though it does eventually.

@Newbieadmin – The best thing to do is download the Nick script. ( since you are a newbie )
Once you download upload the file to your server root directory go through SSH once you are in the terminal window run these commands
cd yourwebsite.com
Then run this command

time php cleaner_2.4.php 2>&1 >> cleaner_log

if you want to see what is going on – type in
tail -f cleaner_log

let me knwo if you have any questions

For all those that are looking for a fix for the malware hack, this is what I’ve used to clean things up:

http://www.php-beginners.com/solve-wordpress-malware-script-attack-fix.html

I’ve been running the CLI version. It does not clean up the hidden .logs directories, so you’ll need to manually remove those.

Be sure to manually inspect ANY .htaccess files for redirects that shouldn’t be there! Additionally, follow the instructions on DH’s wiki about shutting down world-writable directories:

http://wiki.dreamhost.com/Troubleshooting_Hacked_Sites

Once things are back to normal, install the timthumb vulnerability scanner:

http://wordpress.org/extend/plugins/timthumb-vulnerability-scanner/

REMOVE any and all unused plugins and themes.

Finally, BACKUP YOUR SITE! The DH “restore domain” feature did not work for ANY of my sites.

HTH

From the first update: “We apologize for the short service disruption” — SHORT?! No way. I have been problems logging into the panel since the weekend!

@Sparker

Thank you! I’ll come back with questions. REALLY appreciate it!!

March 5th, 2012 at 9:36 pmJoseph Croasdaile Says:

@Sparker will this help in regards to the DB issues?

So, who should I switch to?

All my sites are hacked? The WordPress dashboard is not displaying correctly…is this a hack can Dreamhost restore

@Cogs4x4: [I seem to be in the market for a new vps hosting company.
– don’t tell me GoDaddy
Anyplace else that supports decent mysql and ssl? ]

http://www.arxnet.org/service/aff.php?aff=003

My clients web sites were also hacked, with a change to .htaccess and all PHP files had a trojan added to them.

The site itself keeps its own page access logs in a database. The logs appear intact and do not show any page accesses or errors either before or after the time of the infection, which to me implies the infection was through the server file system and not the web site.

The permissions on all of the folders (including the site root) are correct and only give write access to the owner.

It’s not resolved. Try again. >:|

DH I expect a full root cause analysis with full disclosure by a third party.

@Joseph – Sorry for the late reply – I needed to get some sleep sometime.

I have spent the last 14 hours cleaning up this mess.

So far I have not seen any actual DB intrusion.

Let me know what kind of DB issues you are having. Maybe I can assist. Otherwise it might just be a DH issue…

@Neil please look at the statements I have made above, regarding Nick and Mark’s resolve.

This may help get your clients back up and running in a timely manner.

Chronix – i want a full disclosure that you pay the price of 3 chocolate bars for this service, and that you have done a full root cause analysis of why on earth you would use the host for your incredibly critical hosting needs. Pay up and stop being so cheap if you’re going to be demanding like you are.

Chronix – This is Dreamhost, the only hosting company you are ever likely to see an honest answer from. They aren’t perfect (nobody is) but they are honest and forthcoming with information about what is going on. No host is perfect, but if you want to use cheap hosting for important business then you need to be prepared to deal with situations like this. You can spend $500-$1000 month on dedicated box (that doesn’t scale at all), but eventually you will find out that when bad stuff happens, you are just as screwed and on your own there as you are here. The difference being that here you have an extra $489.05-$989.05 in your pocket at the end of the month. The fact that you didn’t prepare for the inevitable is your fault. You can move somewhere else, but it WILL happen again, and if your not prepared you will be screwed again. Do a cost benefit analysis and see how much this issue costed you, and then go find out how much it would have cost you to prevent it with an offsite backup, dedicated IP, fail-over DNS and a backup hosting plan with another company.

I’ve been using various hosting services since it first became a business back in the 90s. Every one of them has had both big and small problems. Only Dreamhost responds with honest post about what is happening. That is why I’ve been a customer since February of 2006, and plan on staying a customer for a long time to come.

Goofo – Perspective! ;)

@Goofo @Nick I pay well more than the price of 3 chocolate bars for hosting at Dreamhost. I have 3 accounts (2 VPS) which total well over $200 monthly. I have been a member for years and used to recommend Dreamhost to everyone. I chose them because they said they could scale. While @chronix is a bit extreme, this is uncalled for. Also, I have experienced excessive downtime (easily over 40 hours worth since last November). A quote below from Dreamhost’s website:

“Enterprise-Level Reliability

DreamHost manages its own network, so that your site never skips a beat. DreamHost’s network is backed by a 100% uptime guarantee policy and you’ll have around-the-clock (24×7) access to our expert technical support representatives via email, and live chat.”

I think the frustration here is that if DH is going to say they can do something, they better be able to back it.

Why did this whole page get pulled from the main dreamhoststatus.com page? Odd that a major outage like this is suddenly swept under the carpet.

All of my cleaned files were reinfected this morning. Anyone else seeing the same issue?

here we go again Intermittent email

@James – So far the ones I have cleaned have stayed clean.

While I am far from getting to all of em.

2 Things I suggest.

Download a copy of the site. Scan it with your local Antivirus software.
Run the script(s) provided above to clean out the extra BS.

@ Dan – It was not actually swept under the carpet – It was set as resolved. This will auto move the posting to the proper location.

I don’t know why, but my comment explaining the steps I took has been sitting waiting for moderation since last night. Here it is again:

For all those that are looking for a fix for the malware hack, this is what I’ve used to clean things up:

http://www.php-beginners.com/solve-wordpress-malware-script-attack-fix.html

I’ve been running the CLI version. It does not clean up the hidden .logs directories, so you’ll need to manually remove those.

Be sure to manually inspect ANY .htaccess files for redirects that shouldn’t be there! Additionally, follow the instructions on DH’s wiki about shutting down world-writable directories:

http://wiki.dreamhost.com/Troubleshooting_Hacked_Sites

Once things are back to normal, install the timthumb vulnerability scanner:

http://wordpress.org/extend/plugins/timthumb-vulnerability-scanner/

REMOVE any and all unused plugins and themes.

Finally, BACKUP YOUR SITE! The DH “restore domain” feature did not work for ANY of my sites.

HTH

@Jeff – well, I think your choice is clear. And thanks for a post that had information enclosed in it that is helpful. I had never considered it from the point of one of the more expensive packages. I think that might be the problem – DH is trying to be two different hosts at once. Perhaps spinning off the VPS business will benefit both kinds of user.

I am a total admin noob. I know absolutely ZERO, NADA, NOTHING about php, SSH, scripts, or codes (this is why I used WordPress).
Both of my sites were compromised and I’m thinking that I’m totally screwed. Sparker posted info on how to fix it, but the instructions sounded like a foreign language to me. FML.

I got this from DH today- >>”We are currently testing an improved scanning and file cleaning tool. This tool will be checking for malicious content as well as for bad permissions that cause repeated issues with successful hacking attempts. Due to the extra checks and fixes, the scan is going to take longer than usual. Due to the recent stream of WordPress-based hacks, we also have an unusually large number of customers waiting. We appreciate your patience. “<<

Do you think DH will eventually be able to repair my sites by using their "tool"?

Hi Dreamhost,

Still haven’t received a reply from my support ticket from 2 days ago, what gives?

Well this morning the problem is a bunch of images are missing from my site. I will start reuploading them now but can you tell me why this is happening still? They were there yesterday. It appears to be images we uploaded this past Sunday and the Sunday before that are gone. I am on Stampeders by the way.

It would be nice to have at least one day where our site functions normally and images didn’t start disappearing and the site runs quickly and I still would appreciate someone getting back to us. We’ve been customers for a very long time but we can’t promise we will continue to be considering this level of service.

thanks

Website also went down but I couldn’t access dreamhost either to report it…which is frustrating when my business server goes down and customers are calling asking when its going to be up. Dreamhost, you need to communicate better with tangible updates and be more authentic. Your updates make its seem like a single system was having issue yet Dreamhost.com was unreachable and all my websites where gone and timing out when trying to access.

From what I can read from above many are also frustrated by how major the outage was and how non-chalantly you addressed it.

All my Drupal sites are currently down….again I don’t think this issue is resolved. When I requested info on what was done to resolve this the first time, I also wanted to know how it would be handled so that it would not happen again. OBVIOUSLY, that wasn’t done. I will spend the time while my *Sites* (notice the plural there) arte down looking for alternatives to DH, like Bluehost or maybe even Amazon.

WHEN WILL THIS MADNESS STOP!!! ……… @ Where is the support? WHy is there just one guy dissing the DH customers here?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver172′ moving to new hardware and restoring from backups (RESOLVED)
4:01 am

Web VPS host server ‘homie-vserver172′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 5th, 2012 at 4:01 am PST) by jj

Due to complications with the drives on the web VPS host server ‘homie-vserver172′, we are moving the server over to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this situation. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact support with the details.

Please note, this is regarding the web VPS host server ‘homie-vserver172′ and no other services (email, MySQL) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

Update Mar 10th, 1:07am PST: This restore is now complete. Thank you for your patience during this process! Please contact support if you have any issues with your sites or vps server. -Oscar

Update Mar 10th, 1:07am PST: This restore appears to be finished, but we are waiting to call it good until our Admin team has had a look. All VPS guests on this server are reporting online at this time, though! We will update this post again in the morning with confirmation that this is complete. Thank you for your patience during this process! Please contact support with any questions! -Patrick

Update Mar 9th, 7:43pm PST: All VPS guests are currently reporting online and the restore is moving right along. Please contact support with any questions, or if you are experiencing any issues! -Dan J

Update Mar 9th, 2:30pm PST: The restore is still continuing to run and all VPS guests are up and running. If you are having any problems with your VPS, please contact support. – Oscar

Update Mar 9th, 5:50am PST: All VPS guests are currently reporting online and the restore is moving right along. Please contact support with any questions, or if you are experiencing any issues! -Patrick

Update Mar 9th, 2:44am PST: All VPS guests are currently reporting online and the restore is moving right along. Please contact support with any questions, or if you are experiencing any issues! -Patrick

Update Mar 8th, 9:51pm PST: The restore is still moving and we are unaware of any VPS servers having issues at this time. If you are seeing any specific issues with your VPS, please contact support. -Dan J

Update Mar 8th, 5:59pm PST: The restore is still moving and we are unaware of any VPS servers having issues at this time. If you are seeing any specific issues with your VPS, please contact support. -Dan J

Update Mar 8th, 11:45am PST: All VPS servers are reporting up and the restore is progressing without any issues. Please contact support with any questions you have! – Oscar

Update Mar 8th, 5:44am PST: Our Admins are investigating issues with the name resolution for the guests on this host server, but otherwise the restore is continuing without issue. Please contact support with any questions you have! -Patrick

Update Mar 8th, 3:10am PST: Our Admins are investigating issues with the name resolution for the guests on this host server, but otherwise the restore is continuing without issue. Please contact support with any questions you have! -Patrick

Update Mar 7th, 9:47pm PST: The restore is still continuing to run and all VPS guests should be up and running. If you are having any problems with your VPS, please contact support. – Dan J

Update Mar 7th, 5:02pm PST: The restore is still continuing to run and all VPS guests should be up and running. If you are having any problems with your VPS, please contact support. – Seohee

Update Mar 7th, 11:00am PST: The restore is still continuing to run and all VPS guests should be up and running. If you are having any problems with your VPS, please contact support. – Oscar

Update Mar 7th, 5:25am PST: The restore is still continuing to run; all but a handful of VPS guests should be up and running and we are looking into those experiencing issues. If you are having any problems with your VPS, please contact support. – Chalise

Update Mar 6th, 4:30pm PST: The restore is still moving and we are unaware of any VPS servers having issues at this time. If you are seeing any specific issues with your VPS, please contact support. -dan w

Update Mar 6th, 1:00pm PST: The restore is continuing and all but a couple VPS are up, which we are still looking into. If you are seeing any specific issues with your VPS, please contact support. -Brians S

Update Mar 6th, 9:20am PST: The restore is continuing and all but a couple VPS are up, which we are still looking into. If you are seeing any specific issues with your VPS, please contact support. -Justin

Update Mar 6th, 3:22am PST: Our Admin team is working hard on resolving some issues with a few of the guests on this host server and then all will be back online. We will update this post shortly with more details on this! Thanks for your continued patience! -Patrick

Update Mar 5th 9:11pm PST:  Your sites should now all be online again using backups dated 2/25/12. We’re also going to be using those backups to restore files to the new hardware. All VPS guest should now be online and the files are currently being copied to the local file system. We appreciate your continued patience. We’ll update this post again as the process continues. – Dan J

Update Mar 5th 9:11pm PST: The earlier connectivity issues are no longer affecting this server, so we resumed the process of mounting the backup data to the new server. Soon, your sites will be online again from backups dated 2/25/12. We’re also going to be using those backups to copy data to the new hardware. At the moment there are only 8 VPS that need to be recreated, our admins are doing that now so we can start copying files to the local file system. We appreciate your continued patience. We’ll update this post when the sites are online again. – Dan J

 

Update Mar 5th 7:00pm PST: The earlier connectivity issues are no longer affecting this server, so we resumed the process of mounting the backup data to the new server. Soon, your sites will be online again from backups dated 2/25/12. We’re also going to be using those backups to copy data to the new hardware. At the moment there are still about a dozen VPS that need to be recreated, our admins are doing that now so we can start copying files to the local file system. We appreciate your continued patience. We’ll update this post when the sites are online again. – Dan J

Update Mar 5th 3:28pm PST: The earlier connectivity issues are no longer affecting this server, so we resumed the process of mounting the backup data to the new server. Soon, your sites will be online again from backups dated 2/25/12. We’re also going to be using those backups to copy data to the new hardware. We appreciate your continued patience. We’ll update this post when the sites are online again. – Andrea

 

Update Mar 5th 12:00pm PST: We’re currently experiencing some connectivity issues that is causing services on this machine to be unavailable. This incident is unrelated to the recent issues this server has been experiencing on its own. Please keep an eye on this status post and we will update it as we have new information.

http://www.dreamhoststatus.com/2012/03/05/connectivity-issues-to-some-of-our-datacenters

Once that issue has been resolved and the service has been fully restored, we will return to updating this post with information specific to this machine. Please contact support if you have any questions or concerns regarding this. -dan w

Update Mar 5th 9:30am PST: The guests are half way created, the rest will be happening shortly. Once they are created, the restore from backups will commence with the VPS guests running from the backups data. – Justin

Update Mar 5th 7:30am PST: The new hardware is ready to go, the backups mounted, and the guests are being built right now. We’ll update this post again soon with more information when all VPS guests are running. -Justin

This entry was posted on Monday, March 5th, 2012 at 4:01 am and is filed under General Outages, Multiple Server Issue, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

310 Responses to “Web VPS host server ‘homie-vserver172′ moving to new hardware and restoring from backups (RESOLVED)”

The following VPS guests are affected:
ps25910 ps25934 ps26134 ps26160 ps26183 ps26202 ps26291 ps26302 ps26444 ps26553 ps26588 ps26590 ps26707 ps26776 ps27251 ps27651 ps29045 ps29443 ps29463 ps29514 ps29567 ps30039 ps30053 ps30293 ps30305 ps30322 ps30387 ps30723 ps34805 ps35117 ps36310 ps36591 ps37210 ps37504 ps37512 ps37771 ps37853 ps41282 ps44125 ps44128 ps44469 ps47446 ps48098 ps48421 ps48829 ps63117 ps65087 ps65640

Hi,

What about ps55331? I cant access any of my sites? Only through FTP.

CJ: as noted, ps55331 is not on homie-vserver172 so is not part of this issue

Hi everyone, to understand more about what this means, I urge you to read all posts under the threads created when the same happened to the port-au-prince and the castor server…

When will it all end? sell well in Europe because if I do not have translators always translate with google! and you do not understand anything you have an Italian with you that I speak with him? my site and stop by more than 36 hours!!
my machine and vindemiatrix
let me know

You guys have disabled my account for over 6 hours without a) contacting me b) responding to my 8 extremely urgent tickets c) showing any concern for the fact that you’ve crippled my portfolio of 20+ websites on your overpriced VPS servers which in the past month has been down for hours on end!!!!

This is just getting ridiculous. If one of your techs is reading the support tickets and ignoring them this is completely unacceptable!

– Johann

@Tomas: This is completely different than port-au-prince and castor. It isn’t correct to assume that one server backup issue is related to every move to new hardware. The homie-vserver172 server, I just checked, has backups that are less than 9 days old.

@CJ: Your VPS is running and not on homie-vserver172. I checked it, I see ftp running but apache was not. Looking and responding to your tickets.

@Johann: You were written to 8 hours ago by our abuse team. Tech support does not handle these tickets, you must wait for a response from abuse. There is nothing I can do for you at this time. Sorry.

Anyone who bothers looking back on the comment threads each time your servers go down for days, will see that the nature of each incident is always the same – and by following the history via this page, dreamhoststatus.com, we can see that EACH AND EVERY TIME the desperate cries on behalf of your customers are the same and your response – or rather failure to do so – is also the same.

This has been going on, we can see from the thread history, for almost a year!

Anyone can go do dreamhoststatus.com and randomly pick any thread created when a shared or VPS server goes down, whether it was one, four six or ten months ago and see that

a) people lost their work, websites, clients and money

b) Dreamhost claimed it was unexpected

c) Dreamhost failed to answer support tickets for up to days

Now, with this in mind, I have a very valid question: You have already stated elsewhere that you have thousands of servers and that the percentage which fail is very small, which is fine, but knowing that you have one entire server failing for days almost every day throughout the year – with people subesquently being left in panic as they arent informed beforehand – WHY havent you made sure that you have enough staff to deal with these support requests?????

I mean, you KNOW you will have clients crying their eyes out and begging for information EVERY DAY throughout the year, as this clearly and without shadow of doubt IS PART OF your company strategy and organization. Anyone bothering looking through the outage history can see that this is part of how you run your business, you have a server going down like this “unexpected” almost every day, so WHY havent you got people to deal with the reactions?

it has been like this for a long long time, what is your explanation?????

@Tomas: Yes, you can look at previous posts and see issues. What you may see as large amounts of problems, is also openness about what issues are currently going on. Most web hosts will have a comments-disabled status post, or maybe nothing at all, so externally they appear to not have any issues.

Hardware issues will happen, at every host. Some of the recent server issues are our older servers, which we already have plans in the future for upgrades, but it is a big process.

It is and always has been recommended to make proper backups of your websites and databases, offsite, which is best practices. Our backups are being improved as well, for example homie-vserver172 only has 9 day old backups, but it is important to make backups anyways.

We are not expecting a server to fail everyday for the year, it has just been a higher number than usual lately. We are dealing with the support requests as best as possible, and tickets are not being ignored.

Something is seriously amiss at DreamHost.

Usually I first assume that maybe it’s an action or configuration issue on my part, but this has been going on for some time.

For example, pages and other actions completely disappearing as if my site was restored from a previous time.

Or endless server errors or site cannot be reached.

This is happening far too often and without changes on my end.

if tickets are not being ignored, why are they unanswered for hours or even days then?

@taktik(z): I checked your account, there is nothing unusual from what I can see. If you can provide specific details to us, like what record should have been added to the database, what didn’t happen, we can see what we can find.

allora ora scrivo in italiano se avete un traduttore benne seno vi arrangiate !!! il mio sito e ko non vedo Più le immagini i video i commenti non riesco neanche ad entrare perché non si trova la pagina !!! o pagato Più di 60 dollari per avere questo servizio!!???

Arrrggghh. When will this be resolved? 6+ hours of downtime is unacceptable.

ora sonno 39 ore che aspetto!!! ora basta!!!!!!!!!!!

Do we have an update as to how long this is expected to take? I’m a fish out of water without my SVNs up and running.

As for the other complainers – I have some sites hosted on 1-and-1, and they have just as much downtime, if not more. However they don’t have transparency and a nice blog like this to let us know what is going on. You just have to sit with support where they try to avoid telling you what is actually wrong.

@Brad Nix: We are working on getting the VPS up and running as soon as possible.

@Norm: The VPS on the entire host machine should be up and running hopefully within a couple hours. If afterwards you see any issues with subversion data being a few revisions too old on the backup, submit a ticket with my name in the subject and I’ll get that fixed up from the old webserver.

I’m on homie-server173, ps52432. We’ve been down for the better part of 7 days now. Someone rebooted ps52432 on 3/1/12, that’s the last time I saw my website (for about 2 hours, very slowly). Please, please, please, give us some relief. It’s like everyone quit working on the 173 server, and it’s the oldest open problem you have.

Tickets: 4793254, 4793360, 4794345 – First ticket is 4.5 hours old with no response. The others are 16+ hours old with no response. Server still offline and doesn’t even respond to a ping. And…I know, ds1827 isn’t on this server but I’m tired of creating tickets with no response. At least here it looks like someone is responding. Wish there was a telephone number for support that I could call about my dedicated server that has been having issues since last week. =/

How is it my site and e mail is down again. this is so insane to keep happening on Monday’s Move me to something more stable please and get me up and running now as I’m working on a big conference and this is preventing me from working!

My site and dreamhost.com are both down, here we go again..

Looks like dreamhost.com fell into the ocean =:O

My website is also down.

Cmon guys. This is like the 5th time this year we’ve had an outage and it’s only the beginning of March.

Panel dreamhost not work

Is it just me or is http://www.dreamhost.com down? (as with all my sites)

Must be another datacenter-wide outage again like the last Black Monday they had..

Dreamhost Panel is DOWN now. As is our e-mail. Can’t log a ticket to tell you our e-mail is down.

March 5th, 2012 at 10:47 amyou useless fucking idiots Says:

I literally am in the middle of backing up my last website so I can leave this shitty hosting forever before I have to renew it in 2 days time.

YOU ABSOLUTE FUCKING CUNTS

All of my sites are down too. And the panel is down. BOOO! HISSS!

They’ll probably say it is a DOS attack and they will take steps to prevent it from happening again…then, 1 month later…

All my dreamhost sites are down. Whats up?

<3 dreamhost keep up the good work

All my sites are down. Email is down. Panel is inaccessible. You’ve just fallen off the internet again.

I am unable to access my control panel.

Website and Email down again, can’t connect to dreamhost to file a ticket. More consistant uptime would be appreciated. Please get this fixed ASAP.

Thank you!

Now our site is down. No panel to even report it. What a joke

This is more than a server outage. The panel is down too. Please update us with the broader impact and if you know what is going on yet.

All sites are down. Mail is down. Panel is inaccessible.

10:45am PST. Our sites and panel.dreamhost.com are down.

DOWN! PANEL DOWN!

Ditto, site is down and can’t log into the control panel to file a support ticket.

It’s a good thing they have this site up so we can complain in the comments.

The whole freaking mess is down. Again. Perhaps the hamster whose wheel powers the datacenter had a heart attack.

What is going on at dreamhost? This is what the 3rd time that everything has crashed (sites, emails, dreamhost.com, panel etc) in the past what 6 months?

Everything is down, the panel as well.

i have a feeling they don’t even know they are down

The only thing more frustrating than everything going down is coming to this site and seeing nothing to address it

dreamhost.com is completely non-responsive, as is my website. If there is a problem with your datacenters, I suggest looking into different options.

Dream Suck
Again
Again
Again
Again
We will never stop down!

Welcome to shared hosting.

All down here right now.

MORONI (webserver) is down AGAIN !!!
Opsss .. http://www.dreamhost.com/ is down too !!
Crazy !!

This past month has been the absolute worst hosting experience/service I’ve ever dealt with. Servers continue to fail on a daily basis, now not only is my website down for probably the 20th time this year, but dreamhost.com itself is down as well. If a host’s website & panel is failing to load, isn’t that a fair reflection of the quality & service they offer? Looking to migrate as soon as possible, that is if I’ll ever be able to login to my panel and back-up all the files. C’mon DreamHost…NightmareHost.

My site is down.

Whats going on?? this is the fourth time this year without service in my web page, emails and i cannot enter the panel to report anything!!

I have a lot of domains hosted with you and i’m starting to worry and looking for better options.

I think something is not ok in your management of the servers

March 5th, 2012 at 10:54 amTony Pagliocco Says:

Uh whats going on sites are dying left and right as of 1:45 pm EST

Everythings is down!

Terminator – shared hosting has its issues, but this is their entire *network*. Dedicated servers and VPS are down too.

I see a new slogan for them: “Dreamhost.com – making bloody GoDaddy look like saints since 2010!”

Sites down here, too. Panel inaccessible. All during the middle of the work day on a Monday…

Just before everything went down, I noticed that my index.php had a bunch of injected code. A big PHP eval string that inserted a javascript redirect. Since dreamhost is down all over, I’m thinking something got hacked that affected everyone (I assume if just I was hacked it would have been a more targeted, less generic attack)

Everything is now down for me as well… Need a status update ASAP

What is going on? We have 3 different hosting accounts – three different servers – Two different VPS. ALL SITES ARE DOWN.

Last straw… time to start shopping around.

Our site is down and can’t access panel. Again.

DreamHost down. Our email down. We can’t run a business this way

Man, am I glad you people don’t run an airline.

The Panel is down, servers are down, but no update from DH. What’s going on?

Everything is down. Websites, Email, FTP, PANEL.DREAMHOST.COM IS DOWN There is now way to report this except to post it off topic on a forum that no one from dreamhost reads…

Anybody in CA that could stop by and knock on their door?

I cannot access the Panel to see if I am affected. This is down along with my website.

You’re kidding, right? Down AGAIN? This very morning is my delivery morning, and my clients are probably screaming at me, but of course I can’t hear them because my email is down. This is too much, you really are incredibly unreliable.

i think they unplugged everything on the way to lunch

All of my sites seem to be down as well, and panel isn’t accessible. FYI, will watch for updates.

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.” — too bad, because Dreamhost.com is down :-)

@Ben
Then its a total network failure. But thats not that uncommon at DH ;).

EVERYONE MOVE ALONG, NOTHING TO SEE HERE

March 5th, 2012 at 10:58 amNightmareHost Says:

dreamhost.com DOWN!
panel DOWN!
FTP DOWN!

DreamHost.com should switch hosting providers to someone a little more reliable :P

IF Dreamhost spent a little less time worrying about their damned roadshow and more time worrying about their incredibly shoddy product we may be OK. To all those people with the “this is a cheap host…look for something more expensive when your business relies on it” I say…Dreamhost should be held accountable for false advertising!!! Their garbage rhetoric and misleading advertising needs to be brought to the attention of consumer advocate groups and the BBB. Poor job Dreamhost!

You are all very sad people if you cannot live without the internet for a few minutes.

I think Dreamhost is great value.

DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS DDOS

Dreamhost.com is down so I can’t contact you. And my site is down on an extremely critical day. This is the first day of a major conference and a marketing push that just started, i’m losing business and money. When will you be up? What’s the deal here.

This is just crazy. Virtually 95% of what we do to make money happens via email. And, once again, we’re down. The gratuitous “sorry of the inconvenience” that Justin posts on the status page doesn’t lessen the negative impact on our ability to do our jobs and serve our customers. I’d put in another trouble ticket, but I already have one pending on an unrelated but equally frustrating issue from last week. My patience is wearing thin…and I can see the sentiment is the same from so many other Dreamhost customers. Who is the brain behind making system changes on a Monday, rather than on a lesser trafficked day?

All of the sites I am hosting on Dreamhost are down? I can’t even get to my panel! What is going on? At this rate I could host with goDaddy!

Few minutes? It’s not the Internet. It’s hosting. H.O.S.T.I.N.G. which is H.O.S.E.D.

Usually the one to hang in there and cope with the situation but it seems that whenever something needs to be migrated at DH it turns into a datacenter wide meltdown… might fast track the goodbyes for real this time…

no updates on their twitter account about this either

It’d be nice if you all posted about the downtime on the root status site. I’m sure it’s all-hands-on-deck over there, but naturally we’d all like to see some ETA, etc.

A whole lot more is down.

I cannot get to any of my servers through ssh, http, imap. Examples are labelle.org, scottnolan.org, vampyr.org, infinityballroom.com.

I cannot get to https://panel.dreamhost.com to report the outage.

Oh – they are coming back online as I write this… though not the panels.

Today is the day I find a new host and never refer Dreamhost to another person in my life.

Too many problems over the past few months to take Dreamhost as a serious hosting solution for my businesses. I am losing both money and customers as I type this.

Not cool, Dreamhost… Not. Cool.

websites, Email, back up Panel still down

thanks to who ever knocked on their door. I’ll recall the pony rider.

no e-mail. no website. and now, no panel to report any of it to you!

My site is up…

Ahh… marvellous. No internet. Even better, no emails.

Im back up.

Somehow my site is up though the panel and main site are still down.

All things down… all…. this is the last straw, I cannot stand this anymore..

Our mail was working, even when the site was down. Must be something that propagated throughout DH.

Mail, unfortunately, is down now.

This is pretty bad…

I can’t tell at this point if I’m paying for online or offline hosting.

@martin: You’re an idiot.

FYI. Dreamhost’s BBB rating is a solid “F”. Surprise, surprise.

Mail just came back for us.

The ftp is ok, but mysql is what seems to be broken.

My sites didn’t go down, just email and panel.

Everything down…. KIND OF IMPORTANT FOR MY BUSINESS TO MAKE MONEY TO HAVE SOME BASIC SERVICES WHICH I PAY FOR!!!! EMAIL AND MY INVOICE SYSTEM, WEBSITE AND STORES!!!!!

Sites and SFTP back online! Thanx DH! See ya in a few hours, when you unplug some memory from one of the servers, and the whole datacenter is down again ;).

How does one formally report an issue if the panel is down?

Their uptime guarantee only applies from the time you report the issue. Of course you can’t report an issue when the panel is down. Brilliant.

argh!!! my website is down “database access error” and dreamhost is down too!

can’t hit dreamhost.com

That is really bad.

Don’t know,, what is hell is going on there?

Dreamhost.com is down. How to contact you guys. BS.

On the one hand, I’m glad to know that I’m not the only person who completely loses their cool behaves like a schmuck when the servers go down. On the other hand, I’m glad I’m not on the receiving end of it here. Whew!

The fact is that mothman is down, too, at least. That’s where my sites are hosted and there is no mention of it. Plus the panel is down, so you can see how, from my angle, it looks like you aren’t being up front with how large your actual problem is and how many machines are being affected. You patted yourself on the back, Dreamhost, for allowing comments and posting the current status, but you do not seem to be reporting completely and honestly on the nature of the outage.

That having been said, I’ve been a customer for years and I have never received friendlier, or more patient help, than I have from Dreamhost support. I’m a loyal customer and will remain one. But, ah… please tell the truth and please get it fixed, k?

STILL DOWN

It looks like everything is down. All of my sites and dreamhsot.com

Email is down for our domain. Site is still up…

March 5th, 2012 at 11:06 amTammie Dupuis Says:

Dreamhost.com, my business website, and my business email are all down. All of my business is via that website and that email and I’m looking at a busy Monday but can’t do anything about it because everything is down. I’m losing money even as we speak.

There is no way to contact anyone as the method usually used (i.e., dreamhost.com) is down. This is my only recourse.

Please post a status on when we can expect to be back up.

My site is down! Please fix asap

Indeed, panel, dreamhost.com advertising itself, all my various sites are down. Hopefully someone is looking. CAN’t post a service request, as the panel is down!

Okay, website back up, but all of the data in MySQL is gone. Poof.

Email is down for all my sites.

I still don’t understand why they seem to have no redundancy in place. Even when I hosted my own sites, I had two servers in two different locations and I used a service called dnsmadeeasy.com, which – if one of the servers went down – would forward the domain to the other server. It blows me a away that dreamhost seems to have nothing like this. This is 101 server hosting stuff!

All of my sites, just went down – 2:00pm ET, and then came back up – but none of the wordpress sites can connect to their databases. I’ve had problems with those sites, and was trying to migrate them to HTML – but now cannot even get to them. And – I agree with Richard Lucas above – the truth isn’t being told – and some of the issues we’re seeing – I have emails from dreamhost personnel acting like it’s my accounts that are part of the problem. I’m seeing more and more that I don’t think it is.

Everything down! people are dieying please help!!!!

My sites up, panel and dreamhost.com down… WTF?

I’m tempted to add the main site and the panel to my monitoring stuff to see if my sites have better uptime.

Same here, all services down. Can’t even get into the Panel to open a ticket. Booo!

so my site was back up for about 5 minutes, now it’s down again. What gives?

Hey guys,

My site appears to be up, but my emails aren’t coming in or out. It’s cycling and saying it can’t reach the sever. Which is causing it to loop. Dreamhost.com and panel.dreamhost.com are down. I’m on the spawn servers and can’t get in via ftp either.

@myself… “That’s too bad… all sites and FTP are down again. Jippikajee moddafocka’s!!!”

I can not enter the panel

PANEL IS DOWN!!

WEBMAIL IS DOWN!!

Hey, where is dreamhoststatus.com hosted? I think I might switch to their servers?
This always seems to be up! Upon closer examination with who is, its on a Dreamhost server…

EMAILS DOWN TOO!!

We were up for a second, then back down. Mail now too. Still no issue for this on dreamhoststatus.com?

Panel and webmail are down … again!

Website Down dreamhost. Whats going on?

down again??? really? Come on people….

This is like watching Jersey Shore for nerds.

@richard and @jennifer, my guess is that mysql isn’t gone, only that the network is still down.

We have been having slow loading issues for a week or so, and like @Jennifer, had been told that our site is the problem…

Dreamhost sucks. Now I wish I hadn’t re-upped my service for another year.

All my sites are down! Can’t access Westhost to post problem ticket for support.

All of my sites are down too :(

While problems don’t impact me often, when they crop up they are generally huge and with today’s lack of access to anything under dreamhost.com there isn’t any way to report things.

Frustrating that they can’t find the three minutes to post something under dreamhoststatus.com as a way to communicate. That’s why this domain was put up a few years back — to allow for updates when their main domain went down.

I don’t expect five 9’s (but I’d like it!), but updates, please, DH?

My stuff is flaking out, too, and I understand about being angry when a problem like this jams up your business. It’s happened to me. But to the poster who posted under the name “you useless f-” etc. … I didn’t do anything to you, but I still had to read your all-caps spew. This isn’t your private channel for venting, and your foul-mouthed incivility is unpleasant to experience even as a spectator. You’re posting in public — have a care for the rest of us.

I imagine if you were to politely contact Dreamhost and explain that the downtime has kept you from making your backups by your deadline they would work something out with you. But gracing my morning with a brisk wakeup of screaming misogynistic cussing ain’t likely to do you much good.

@Karl – I don’t think that approach would work with data-driven sites. I’d love to hear other ideas for redundancy, or recommendations for a new host.

Email is back and working for the moment.

panel.dreamhost.com is down

all my virt servers are down.

whats going on!!!!!!!

What the hell is going on?????

“Duplicate comment detected; it looks as though you’ve already said that!”

Why can’t I say the same thing over and over again… whilst DH keeps going offline over and over again?!

March 5th, 2012 at 11:14 amNightmareHost Says:

martin Says:

You are all very sad people if you cannot live without the internet for a few minutes.

I think Dreamhost is great value.

@Martin – what the he11 are u talking about? This is part of the problem with DH now…DH used to be hard core web programmers, now it is made up of folks like Martin saying things about the “internet” – really, WTF??

this happens like once a month now. we should start getting refund checks at some point for all the down time.

Great. . . Email is not working

Can’t even access the DH Panel, my clients VPS are down.

Everything is down. Website down. Panel down. FTP down.

This is second time in a week that this has happened to my VPS! Seriously!?

The response I got last week from my CRITICAL ticket (the day after) was that there were no errors or problems and that the site was online. Really?! The site was down for an entire day and you have no record of that anywhere?!

@Justin. Thanks I would appreciate if your abuse team could respond ASAP. Any clue when they will be looking into this. They’ve wrongly accused me of phishing and now my 15 ecommerce sites (including multi-million dollar booking systems) have been down for 12 hours. This is really serious guys. The least they could do would be to send a quick message. I’ve sent almost 20 since last night falling on dead ears.

But now I see that dreamhost.com is down also so perhaps this is just the end of the world?

Thanks,
Johann

even dreamhost page is down

Everything’s broke and no post from DH…WTF!?

I have several vps accounts serving up wordpress and drupal sites. — all dead
I have a shared hosting account serving vanilla html sites. —- all dead
My panel is dead
“The connection has timed out
The server at panel.dreamhost.com is taking too long to respond.”

I am a little concerned.

The connection has timed out

The server at http://www.terminatorfiles.com is taking too long to respond.

The site could be temporarily unavailable or too busy. Try again in a few
moments.
If you are unable to load any pages, check your computer’s network
connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

Everything is down. I just renewed. I’m going to cancel, this is junk.

March 5th, 2012 at 11:16 amyou useless fucking idiots Says:

And yet always this website remains working perfectly.

just to put some perspective on this, what about the poor people in syria?

Whew, I almost sent out my monthly newsletter to my blog last night. Glad I waited, otherwise a couple thousand people would be getting (a) html emails without the images or styles, and (b) links to articles on a site that is dead in the water.

I understand these things happen, and it has been more than a year since my server was offline, but at least an update on the status blog or a kindly email to my personal account would be really nice.

It would make these outages a bit more palatable.

Ok. Dreamhost has fallen off the planet again, I have clients calling, and dreamhoststatus.com doesn’t even have a mention of what’s going on. I can’t even get onto dreamhost.com, much less the panel. Last straw.

Can we at least get an update on the panel? And a way to officially report an issue (like the database being down) if we can’t reach the panel?

Who just lost their job? We all want to know…

Guys this is happening way too often.

March 5th, 2012 at 11:17 amUnhappy Customer Says:

Dreamhost,

Please STOP BLOWING IT.

Signed,

Concerned PAYING Customers

Wow… its back up… lets see what F5 does.

Dammmmmmmit. ALL sites down, as well as panel? Guess we can look forward to another month of no live chat assistance as well? Come on, guys, I’ve been a customer forever!

Everything is down. Can’t get to Dreamhost — timeout with tracert. The domains I host don’t even get that far — just ‘not found’…

My site on jbplayer is now down, as is access to the control panel. How am I supposed to get in contact with you if the panel is down?????????? ARRRGGGGGHHHH

shoot yourself.

My sites were back for 5 minutes and now they are gone again. Can we have a status update? Bueller…Bueller…Bueller
You’ve been an amazing host up until the last couple of months. Please make things right, I really don’t want to have to pack up and move!

I would gladly pay a couple dollars more per month for better uptime and redundancy.

DH’s prices have gone down from when I signed up – charge more, and buy more hardware please.

Can’t even get to the web panel, what gives? I’ve lost so much business just this morning!

site says database down; panel.dreamhost.com down, can’t report the error. WHAT IS THE ISSUE and STATUS PLEASE!

Thanks are back up for us for the moment – hopefully a good sign for everybody else. Posting because DH isn’t.

dhstatus on twitter says datacenter connectivity issues… eta on a fix, please?

what the holy crap! no report here of any outage and yet everything is down.

Thanks dreamhost. I have clients going crazy and this is the second time this has happened in less than a month. What on earth is going on here??

Sorry, we’re on it guys. Status post going up momentarily.

The worst part is they still haven’t made an official statement. Jesus guys, at least tell us you’re aware of the issue. This is terrible customer service.

Wow this is insane. Where is everyone moving to! I have 15 sites and can’t access dreamhost or email. Sad

please post an update – many sites reporting database error, my email isn’t working, and i can’t access the control panel. my clients are starting to freak.

at least a status update is in order.

Good luck to you in re-esatablishing all that is hopefully only temporarily lost. Communicate with us as you can. We all realize issues can come up that create a panic situation. I for one will go outside and enjoy the lovely weather.

Keep us posted on your and our recovery.

malditos ladrones de dreamhost

March 5th, 2012 at 11:22 amUnhappy Customer Says:

So where are you moving your websites to?

Booooooooooooo!

Thoughts of the Dreamhost Founders:
“I have an idea! Lets create a hosting company and a shoe string budget. Let’s not worrying about making our hosting fault tolerant. We can just hie behind a crappy support web from when things go wrong. Better yet we can charge extra for a call back support service, but never call them back. LOL, our customers will be a bunch of suckers! You know, one more thing, lets make it really really easy to install crappy canned web site software prone to getting exploited.”

they just did a status update

The panel is down. Emails are down. My clients are crazy. Please hurry up!

OK…what is plan B…I’m dead…What can I do to protect myself from this type of stuff…Monday afternoon really? How can I keep the Titanic from sinking? Anyone out there with some advice for a novice who was on the verge of big things (from a little persons perspective).

Hardware fails. Networks and systems grow more complex. What might have been a smaller problem in the old days can propagate quickly and become a huge problem for everyone. There is no perfect world, or perfect host, and there may not even be anyone to blame, so checkity-check yourselves. For the couple who have bragged about how they are losing millions from a little downtime, I would ask why the hell you aren’t putting some of that money back into your own infrastructure, like a dedicated server?? Shared and VPS is for chumps like me, not for rilly big and busy Web tycoons such are yourselves…:)

Wow. Another outage. I don’t know what the deal is with DH. We get good reliability for a couple of months straight and then from one month to another, it seems the outages are a weekly occurrence. This is exactly what caused us to migrate from our previous host. This is absolutely ridiculous. No website, no webmail, no SMTP, no panel access. Can we get some stability here please?

Update: Website is up. Panel is still down.

Clients calling me while I am vacation, fantastic… thats it- im gone next month… will slowly move all accounts to a new system.

Please let us know when this is going to be up, I need my site up asap!

I do notice that it seems to happen when I get the who is reminders… not sure why… but looking back when I get those reminders there is often some problem…and outage…

The only comment I have is this: after being a customer for 5 years and with all the latest outages and security problems I have had to deal with across multiple of my domains hosted on Dreamhost, the only option left is to look at alternative hosting… What a pitty…

AGAIN PANEL.DREAMHOST.COM IS DOWN.

DBS AR DOWN

EMAILS ARE DOWN.

JABBER IS DOWNN.

SITES ARE DOWN AS MYSQL HOST IS ALSO DOWN.

WHAT MORE I COULD EVER EXPECT FROM NIGHTMARE HOST.

THERE WAS A TIME WHEN I THOUGHT OF YOUR GUYS MOST COOL.

NOW I FIRMLY BELEIVE YOU ARE PATHETHIC LOOSERS.

PLEASE CONVEY MY MESSAGE TO YOUR CEO

STOP SELLING HOSTING AND START AS PIZZA DELIVERY BOY, MAY BE YOU ABLE TO DELIVER IN 30 MINUTES….

wow?? never seen it this bad on dreamhost, by far my favorite hosting company. i dont see any update or status about this issue even taken place. if and when you get the time could you let me know when we all can go back to work? You are appreciated even on rough days. Good luck guys and thanks for all the good days

Email is back

Dear Dreamhosters,

We need to know when our websites will be down. Even if you aren’t able to give notice well in advance, please give notice immediately when you know who is going to be affected and for how long.

By email.

To those of us affected.

Also, I’ll be getting in touch directly about getting a refund for this month’s hosting. Not that this barely covers the losses of servers unexpectedly dropping all my websites right as a big affiliate email has gone out.

Again, I think most of your customer frustration could be avoided with *advance notice* about outages.

Thanks,
Daniel

I came here to find out what might be happening at Dreamhost, but all I found was the ugly side of their clientele. While I’ve seen a few comments that were well-formed, the immaturity here is staggering. I see why most hosts have comments disabled, and if I were @Justin I would use this thread as an argument to management to disable comments on dreamhoststatus.com to follow suit. Dreamhost could be paying one or two people to try and read this constantly growing thread and make attempts at cordial responses that would be immediately devoured and belittled by an angry mob, or they could pay people to work on their issues and get/keep things up and running.

I’ve been a loyal Dreamhost customer for years and will continue to be. Whenever I’ve reached out to them with questions I’ve gotten prompt and helpful responses. Most of the comments I see on here don’t deserve responses and I’m sorry that Dreamhost staff will likely have to waste time reading them.

March 5th, 2012 at 11:27 amNightmareHost Says:

redundancy anyone?

Seriously this was the last straw for me. I’ve been a fan and tried to promote DH through the problems but fuck this mess. Ever since MySQL became shit here and all of my Joomla sites were slow loading and sometimes timing out I have debated on shopping around. Today does it for me. Its been well over an hour and no reply yet as to what’s going on.

Soon as MySQL is accessible again I will be transferring off.

@Eric: I do not agree. If I had no way to vent and to express my frustrations, I would be far more tempted to just take my ball and go to another host.

well, actually, for what its worth, we had a very expensive host for a number of years called verio I think – and although many times the cost of dreamhost it was much, much less reliable, with many hours of downtime. so there.

We are down too. There really has been allot of aoutages lately. I would imagine it is causing allot of accounts to leave DH.

March 5th, 2012 at 11:33 amNightmareHost Says:

@Eric-

Ok, please share with us your “referral code” – I can tell you are one of the monkeys…..although, we do agree on one thing, I wish they would quit with these “updates” crappola & simply work on FIXING THE PROBLEMS…this on-going “diary” of “How NOT to run a hosting business” is getting really old…

it seems EVERYTHING is down even the Dreamhost site could we just get an idea how long it will take to fix this?? Its better to be honest and not act like Sony does when this happens, because you just get angry customers calling all day and night. besides sony is a gaming system theses are sites we need to make money so it becomes much worse.

but I’m not a monkey. OOOH! AAAH!

As someone who works in the tech department of a small to medium sized company I understand that sometimes things that are our of your control happen. For example, I’m getting multiple calls and emails about the sites, project database etc that my coworkers cannot access due to DreamHost being down. So I sympathize with DH (certainly the employees, who likely don’t have any pull in determining how fast old servers are replaced or how many the company hires to speed up the process).

That said, it does seem that the amount of times DH has gone down completely is starting to get a little ridiculous and people like me have to start considering if I’m doing my job if we continue to stay with the unreliable service.

Also, I very much appreciate the higher level of transparency here than at other hosts, but what’s weird is that it doesn’t always seem correct. Clearly this current outage is system wide, while employees only confirm a handful of servers to begin with. So transparency really isn’t that helpful if it is wrong, totally incomplete or in the worst case scenario, lies.

Like I said, I’m sure employees are scrambling and working very hard right now but you should tell whomever is in charge that it really is starting to become unacceptable. It’s just not good business to save a little money by putting off a server upgrade or not hiring enough staff if you’re customers are this unhappy.

@Richard I must say when I was talking about well-formed comments you were at the top of the list. My first instinct was to laugh at your need to vent, because if you were one of the people just typing profanity or using horrible grammar to encourage Dreamhost to take up pizza delivery I would’ve said “Go ahead, I’ll be glad to remain a Dreamhost customer while you move your data load elsewhere” — but I completely respect where you’re coming from and it’s good to know that these sort of unique features keep people happy.

If I were going to contemplate leaving Dreamhost today it would be out of embarrassment to be grouped with this rabid mob. Thank you, Richard, for being one of the exceptions in this case :)

Wow, okay, sorry folks I have no affiliation with Dreamhost other than being a happy paying customer for more than 4 years.

Looks like everything is down for me. I don’t rely on my website for money though so I’m okay.
Though, I must laugh at all of the complaints. Talk about first-world problems. “Oh noes my internets is down! Whatever shall I do?”*

*This only applies to those who do not make money off of their website/email. I do have sympathy for those who rely on their email/website for their income.

Alright you need to understand that people have a right to vent. They are paying customers and they could be very well losing business because of this. And if you really keep tabs on this you would know they are down a lot. There has been may issues. However, I would agree name calling and cursing is uncalled for.

I have never had another host so not sure if there is anything better. However as a small business owner I often do get frustrated with DH and the downtime. I am glad that have this page though so I am not totally left in the cold.

I have considered looking around before and I don’t know why I keep putting it off. Anyway c’mon DH get cracking, get us back up please !!

Oops, as soon as I posted my comment my website and email are back up. Thanks guys :)

Vitae- you lucky sun of a gun

panel.dreamhost.com is not working and our email is down …. WTF!?!?!

I am still down, and I am one of those guys who does generate income from our site, so I am loosing money as we all sit here and wait. Not to mention my entire staff is sitting around and unable to work.

March 5th, 2012 at 11:58 amyou useless fucking idiots Says:

@eric – “I came here to find out what might be happening at Dreamhost, but all I found was the ugly side of their clientele.”

Yes, this is the ugly side of people trying to run businesses, earn a livelihood and who have paid for a service only to suffer outage after outage after outage without any response from “support”

Dreamhost got greedy, plain and simple. They tried to get more money by cramming as many people as they could onto their servers, going waaay beyond capacity.

Y’know this is actually a microcosm of what’s going on in the greater world. Services and standards get worse and worse every day, but we’re still expected to keep on paying and just take it.

Fuck that.

My hosting comes up in two days time, I was actually rushing to liberate my last site and move it over to a proper hosting company when this shit happened.

I’ve been a loyal customer for years too. Even back in 2008, after they accidentally billed my credit card (and thousands of others) for $500 USD with nothing but a “Doh!” for an apology 4 years ago, I stood by them.

But trust me, everyone has their limit, and DH has gotten to the stage where it tests the limits of their customers on a daily basis.

When you say you’re going to remain a loyal customer @ DH that says to me you’re one of the following:

1. You’re an undercover DH employee trying to make us feel more irritated than we already are

2. You’re a troll

3. You haven’t reached your limits yet, but you will, you’re with them four years, I’ve been with ‘em SIX.

4. Or you’re just one of those people out there with ZERO standards whatsoever.

Personally I think it’s a combination of 1,2 and 4.

But I could be wrong, in which case I wish you all the best of luck with Dreamhost in the future, with your upbeat attitude and their sterling service you guys are made for each other.

March 5th, 2012 at 11:59 amNetrapal Singh Says:

Dreamhost Team :

Are your all server so much Old and you all are working on that. Why do you not change that all. You are not now good service provider. You have to shut your this business and go home. you have to enjoy
at your home, you do not need to do work unless you are making our business so least.

WE ARE LOOSING OUR BUSINESS DUE TO THIS DOWNTIME AGAIN AND AGAIN IN A MONTH >>> ARE YOU LISTENING US.

Our site, panel, and mail were down. Then the site and mail (sorta) came back up, but panel is still down.

Autoforward addresses seem to be either down or heavily queued, and I’m having SMTP issues on one mail account, but that may be a settings issue. (that I can’t access panel to check on.)

But Dreamhost.com seems to be down for me, too.

@you useless fucking idiots:

Does that usually work for you out in the real world? Do you try to think of how to best insult people who are clearly not even part of the problem simply because they don’t agree with your coarseness, immaturity, and rigidity? And then, do they suddenly see the light and say “Gosh, you useless fucking idiots, I guess I am a useless idiot!” and go and do their best for you?

You are having a temper tantrum. Could be related to narcissism. Could be that you don’t interact with real people often enough to remember how to do it well.

Oh man, and I thought last month’s DH newsletter was as humble as it could get.

OK…so much for the “love” among the passengers on the boat…has anyone heard anything at all about the cause for this outage? Has anyone gotten through to a human voice at DreamHost? Any tangible intelligence out there….

Problems with my VPS being down for hours at a time have been reported for over a week, starting on Feb. 26 when DH said it had been rebooted in “read only mode” and no explanations thereafter. Support tickets have gone unanswered for days. Why were the problems with this server let go for over a week before a crisis hit? Now they tell us (3/5/12/ 12PM PST) “This incident is unrelated to the recent issues this server has been experiencing on its own”, which means there are more problems to come that have been put off. The problems with the server are not “on its own”; they are failures in maintenance and priorities.

My websites and mail have now been down for over twelve hours in this latest incident alone. The original status report claiming “no other services (email, MySQL) or servers are affected” is not true.

I’m sure that the individuals now working on the problem are doing what they can, but it’s too little too late for management’s misallocation of priorities and resources.

oh cmon.. what’s happening to dreamhost! O_O I can connect now, but database is still down. :D

I can’t even connect to panel.dreamhost.com. Weird.

A friend who uses a different company just said: “what? thats odd, outages should be pretty rare. My company once had one in 2006 when they had to migrate one of their servers.” Sitting across from him typing this and expressing a tight fake smile….

Get a load of this silliness…

Our Reach Out & Touch You Tour is making its way across the United States stopping in the six cities that, worldwide, contain the most DreamHosters.

You’ll have us all to yourself for a few hours – It’s like a dream come true!

http://blog.dreamhost.com/2012/02/28/let-us-touch-you/

Will look forward to talking to them…

Our database server is down again. What is happening? 4 outages in three weeks.

Can you please check psmysql26138?

Panel is down too.

Thanks Justin for the quick response. And checking my account.

Besides an abnormal amount of server errors, several pages and settings are disappearing.

If my account is sound, it obviously eliminates an unknown quantity; so I’ll continue to troubleshoot accordingly, and document the errors and issues. If more arises, then I’ll be sure to notify you.

However, I do get quite a few site down notices. At least, every other day; and often once a day.

Justin, Can you please look at our server too? It is psmysql26138. It was down on Saturday and now again it is down.

Thanks Justin. It seems to be back up.

March 5th, 2012 at 1:32 pmyouuselessfuckers Says:

@Richard Lucas

Are you the Sigmund fucking Freud of Dreamhoststatus or do you just have a lot of time on your hands?

So, if my email just came back after hours of being down and there are no message over a 4 hour span, but I was expecting important emails. Should I expect them to arrive eventually or are they gone forever? If I alert my clients that my email was down AGAIN, I will look unprofessional. If I don’t respond to emails sent to me because they never arrived in my Inbox, is even worse.

Please advise.

yea, so still no idea when we’re going to come back due to these unannounced server moves, not to mention to general blackout you’re having.
meanwhile plenty of time to look around for alternatives (that’s nearly 10h down now, BTW…)

After this storm has blow over, I’d *really* like your CEO to explain to us what has been happening over the last few months at dreamhost, and why I’m now having to recommend people move their websites away…

@ “you useless fucking idiots” (commenter above)
Six years? Try nine years.
I’ve been with Dreamhost for 9 years.
Maybe I have an infinite amount of patience, or maybe I’m just lazy. (Probably the latter) I don’t plan on switching.
Then again, I don’t make money on my website. I use my website for fun, and email, and personal stuff.
Because of that, I am not effected when they are down. I just wait till they fix whatever needs fixing.

If you rely on your website/email for income, any outage is damaging to your income and clientele. Keep in mind though no one is perfect, all web hosts have issues every now and then. If you feel this one has too many issues, pack up and move out. That’s right, get up and go, find someone more reliable.

If you don’t rely on your hosted website/email for income. Deal with it.

If you think I’m a “dreamhost spy” or whatever, check my Facebook page (You can find a link at http://www.vitaebat.com) and you will see I have no affiliation whatsoever with Dreamhost other then being a customer.

PS: To Dreamhost (if anyone is listening).. Why not offer a bit of compensation for those affected, such as a free month, or some upgrade? It may help a bit.

^^^ Whoops. Looks like I miscalculated. I have been with Dreamhost 6 years, not 9. I thought I signed up in 2003, but it was 2005. Boy I feel silly.

March 5th, 2012 at 2:57 pmJohn Burwell Says:

For what I pay? I have no complaints.

Better to have DH guys sweating over keeping a server running than to have to do it myself. And they always seem to put everything back into place just fine.

The way I add it up, I come out way ahead on service and time. I’m honestly not sure how DH pulls it off at all.

As always, good luck.

We are still down.. Support issue 4795372.. Almoderlig 24 hours.

Hi guys

I dont want to bicker about the problem as it obviously doensn’t work or help the current situation.
You have mentioned yesterday that the issue would be sorted in “a couple of hours” it has been a day now. 24+ hrs is def not a “couple”
Can you please be specific as to when exactly the issue WILL be resolved. I cannot keep promises to my clients made on what you tell us on you blogs and site.
Please I need answers and decisions will be made on whether we need to move or what. I realy cannot afford this as a still new growing company.

Will appreciate your feddback.

Jaco

Everyone of our site on our VPS have been down for well over 12 hours. We have approx 100 small businesses hosting through this and every one of them has called me at least once. Please can you have a look at our account and sort it….. begging even.

It’s been well over 24 hours for me. No email, no shell access, no web access. The sites are not “now all online again” as claimed over two hours ago, it is not “soon” after 3:11 PM PST, and “services” did not “remain online for the duration of the restoration as originaly claimed.

PLEASE look into our issue; 4796177 – we are now over 25 hours downtime.

Hi Guys

all my domains are still down even though the blog says everything is sorted. So unless I have been pased over or forgoten then not all domains are up.

PLEASE ASSIST.

@Will: I am sorry for the confusion, as this issue is not resolved and the restore of data is continuing. When this issue is actually resolved, the post will be marked (RESOLVED) in the subject line. Please contact support through your web panel with details about your issue and we will be happy to help!

Hi!

ps44966 my server is not on the list of those affected, and now?

ps26291 is still down. Over 24 Hors down is UNACCEPTABLE. Arrgghhh. I feel hostage to DH. Please let me pay the ransom and let me go.

We are nearing 30 hours on our server.

The backup that’s been restored on our VPS is *very* incomplete. Missing lots of directories.

* Are you still in the progress of restoring the backup?
* From which date are you restoring?

FYI, for others who have transactional databases that are affected.

My site captures daily transactions from users in a database.. Once my VPS comes up I can restore the database from an offsite db backup to the night prior to the crash.

However, when dreamhost migrates my host to the permanent hardware, they will be using that same february 25 backup. I’m concerned new transactions will be lost. (you think users are annoyed now… Would hate to lose data a second time, and so soon)

I’m hoping it is possible for dreamhost to shutdown and take a current backup of the VPS prior to migration to the permanent hardware.

Just in case anyone was in a similar situation, this is something to be aware of.

John

Why are there no updates to this what so ever!!!???

Really… Now we’re deleting an unpopular blog post entirely?

Update last read all was right, yet still many are offline completely.

Our vps is not desponding to anything http/https nor is ftp / ssh working. But we are able to submit tickets and not get answers.

Again, others have recommendations for another hosting company for vps hosting (other than godaddy?)

Everyone gets hacked sooner or later. All things considered you can’t beat DH.

@John: Mysql databases are on entirely different hardware, no webserver restore will have any affect on any database, unless your software makes the change.

@Sune: This post was updated a couple hours before you posted, it is receiving from 2 to 3 updates a day, more if there are any issues. We can restore from backups for this vps, I’ll check for open tickets and reply with what I restored.

@Cogs4x4: We do not delete any blog posts, if it is resolved, it is un-sticky’ed and may appear below the sticky’ed posts. As for your VPS, you are on homie-vserver191 and not affected by this outage. I will look at your tickets for your issue.

@joe: Thanks! Our abuse team is hard at work everyday to make new protections for specific attacks and are starting more proactive approaches for notifying customers of issues with their sites, to hopefully prevent issues before they occur.

Somebody from Dreamhost actually had the nerve to respond to something in here..

We are on ps26553 and we get the following error: Your home directory does not exist on your web server.

We have been without our server for 31 hours..!!!

@Sune: I just responded to you, all of your data is restored.

@Joel: Your VPS is back up, responded to you.

@Brad Nix: Your VPS is back up, responded to you.

@Justin: At the time I wrote my message we were still seeing the error message I posted in the reply to you.

@Justin can you please give any updates for ps26291 ?

Do I have any other options other than wait on DH?

Why did you remove the Network Outage thread from yesterday??

No explanation of what happened or why??

@Sune: It seems you have an unmanaged apache. The data is there, but you will need to customize it how you wish. I copied over our example config file so it will load for now, there is nothing further we can do for your custom setup.

@Brad Nix: I am working on it. All should be up right now, I’ll reply to you shortly.

@Dave: It was not removed, it was un-sticky’ed. Please look further down the blog, past the sticky’ed current issues. At this time it is still being investigated, I have no information I can provide you now. If there is more information we can provide, it may be provided via the company blog.

What is the status? All admin pages are running errors. Three wordpress sites loading errors. Appache services restarted? Can not afford more down time. Have you rebooted the server?

I am sure there are others who are losing business and at critical times (for me as I launched a web site March 1st).

status?

@Kekoa: You are on tobias webserver, this is not a server having any outage right now. Your issue is specific to your sites and will need to be handled individually.

My url has been down for almost 20 hours now, :-/

I placed another support ticket today about 2 and a half hours ago, still haven’t heard anything back.

The dedicated service accounts have been a bit too spotty lately for the premium price paid

@Jeff: Your ticket and callback are currently being looked at by a support person.

@Justin – if this is specific to my sites, why would I be told all my sites were compromised? WordPress is pushing errors out showing apache errors (from research). Please explain 1) domain completely wiped out 2)Fresh install of WordPress installed 3) NO Script, Coding or anything from any so called compromised web sites used or posted on the ONE CLICK install are displaying the same errors? Could it be dream host was compromised and installing compromised software? Need a little more information other than “your site seems to have been compromised, moving your ticket to our abuse team” more then an hour ago. You would think compromised = immediate action….????

@Kekoa: Please look at any of your php files, such as wp-config.php. There is injected content, causing errors. Tech support cannot help with hacked sites, our abuse team must, so if you haven’t already you should be hearing from them soon. This issue is due to the software you are running either being outdated, or permissions of your directories too open which allowed altering the site that way. The email you will get from abuse will detail what they found. Sorry but you will need to be patient for the results of the tests, there is nothing I can do to speed up this process.

and yes… Sites <—– plural. Dream host has their hands full. And, according to the ticket and statistics on my web pages and where I am hosted, I am on the same server as all over the above issues.

@Kekoa: Yes, if one of your sites on a ftp user is having the issues, all sites will likely be affected and must be cleaned up. I disagree still that your server is affected by any of our status posts. When I go to grabitsavings.com, I am directed to the tobias webserver. This server is not being restored from backups, or having any ongoing issues.

@Justin

Awesome! Thank you guys! I understand everyone over there has their hands full; but with our print marketing and phonebooks going out in 3 days, having this site up and running asap is crucial.

Now I am just waiting for a response from them, but relieved they are looking over my ticket.

Hi, Justin.

My VPS does house a transactional Postgres database, not MySQL. As such, I did not select the MySQL VPS, and consulted with dreamhost support at the time regarding the VPS type selection.

I have a standing support ticket regarding this issue: #4796606.

John

@Justin

I couldn’t wait around much longer, and managed to fix the issue myself. I withdrew my tickets, however if they could just glance over my account just to make sure everything looks as it should, that would be much appreciated

Our site is down! What’s happening, can someone look into this?

ps85428 is offline.

we are ps81981 is down for 2 hours! Whats happening?

@joe & @ ogp: Both of your servers are not hosted on this VPS host machine (you’re both on homie-vserver24). That server became unresponsive and needed a reboot. Both your sites are now loading.

Thanks @dan! u da man

ps26202 has been down all day, I have submitted 3 help requests and have not heard anything? Just wondering if our site will be back up tonight?

@Jack working on your VPS now

Is Dreamhost having any virus attacks on servers? I have a client’s server with multiple sites… I originally thought there was a virus in a WordPress plugin, but everything was removed and security measures were taken, and now there are instances on other sites on the server.

you will upload files or I’m going to have?

@April

YES they *must* be.
We keep cleaning and malware keeps returning.

@April & @joe: No. The most common form of compromise is from outdated software, plugins, themes with known vulnerabilities, or leaving directories open (777 permissions) that can allow adding of data. If you wish our abuse team can scan for any outdated software or known vulnerabilities, just submit a ticket asking to be scanned. For Joe, if they are returning, then you have missed something. All sites under a specific ftp user will generally be infected, so if the ftp user contains multiple sites, cleanup of all sites on that user will be necessary.

@Douglas: The data is being restored from backups and should be available. If you notice missing data, please submit a ticket and we can see about recovering it from the old server.

using python here. all is always ok.
you guys should add cloudflare as an option in DH so we can use the DNS and security provided by Cloudflare as well as the speed up.

@all check out cloudflare.

@Jazz: Keep an eye out for DH + Cloudflare. We should be announcing this sometime soon in our newsletter.

Thanks for the info, Justin… Any idea when the ticket queue will mOve? I’m going on 24 hours without a response from support (including not receiving a call as requested).

Email has been down for hours and giving senders bounce backs. I can’t log into the control panel either to submit a support ticket. Been with ya since 2004 and I’ve never had this many outages. I know it happens but it’d be nice to have an update.

@Sarah Murphy: Please contact Support regarding your issues, as the site you listed is hosted on the “tigers” server and not this VPS host server at all. If you cannot make a connection to the Web Panel either, you are more likely encountering a local network issue, as we are showing no issues with that at this time.

@April: Our team is working their way through tickets on these issues as quickly as possible, sorry for the delay!

Our website has been down for several days which also means our email has been down for the same period of time. The incoming emails are getting bounce backs.
We are a small company and need access to our emails to keep our company in business. I have subnitted a support ticket but have not heard anything back whatsoever.
Please confirm when our website will be back up and running.

Email sent from Masseria shared server not being sent out. 4 hours later and still waiting to receive those emails! Have posted a support ticket but no response as of yet. No mention of any email issues at all on their status site. Does this status site only get updated retroactively once the problem has been fixed, but not when the problems are actually happening?

@patrickm This is not a local thing. I have people in California, WA and ID trying to access http://webmail.clearmindgraphics.com and http://webmail.delmarpiano.com and many other webmail accounts you host and it says the URL can’t be found. No one has gotten email for 16+ hours and I still can’t log into the control panel to submit a support ticket. For you to say there isn’t a problem is a HUGE issue! That I can’t access my control panel to contact you is a HUGE issue. That clients haven’t been able to access email for 16+ hours is BEYOND and issue. Please help!!!!

Oh yea, and all email to contact@clearmindgraphics.com plus all the other email addresses we host is giving people a nasty bounce back saying to check again because these email adresses don’t exist.

Please email me at clearmindgraphics@me.com to help us out with a solution.

I have a hacked site that is sending out a redirect to a PORN site. Since we’re a school, this is embarrassing. I’ve had malware problems for the past three days, but this started this morning. I have an open ticket from yesterday and one from early this morning but still no response to either one. How on earth do you get through to customer service (insert laugh here)?? This is really becoming unacceptable!

@Pat: I also had a dreamhost account hacked by the same thing. Dreamhost didn’t do anything except provide instructions on how to back up to an old version of the website. It cost us so much time and money. I’m very sorry to hear it happened to you as well.

@Kevin: Email is not affected at all by the VPS restore. Email is hosted on a completely separate server unless it’s mail routing from your website. Can you let me know which email domain you are having issues with and I can check it for you?

@Wanda: Email generated from websites is being delayed a bit due to a spam run affecting one of our relay servers. Our admin team has already resolved the issue and mail is flowing now. It may just be a bit longer before the email is received.

@Pat: Your tickets are being addressed by one of our security/abuse members at this moment and should hear back from them soon.

one of my sites was hacked too recently

@Tomas Please submit a support request. This will allow our Security Team to scan your account.

websites are up, but my ftp access is down :/

@Laurence: your site is on the stampeders server, there was a problem with the FTP configuration for that server, we just fixed that

Our site is down and we cannot send or receive emails. This has been
> occurring since Saturday. We understand from reviewing your posts and the
> blogs that DreamHost has had some serious issues which is understandable with
> computer systems.
> Not getting emails is affecting our business dramatically.
> Could you provide us a estimate as to when out site will be back up? We are
> assuming when that happens, the emails will be back up also. Is this a
> accurate statement?
Our website is Stixnrign.com

@Kevin Wilson: Happy to check this out for you, but that domain (stixnrign.com) does not appear to be valid. Please contact support on the “Support > Contact Support” page in your Web Panel and we’ll have a look!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘windhoek’ moved to new hardware and restoring from backups (RESOLVED)
1:34 am

Shared web server ‘windhoek’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 4th, 2012 at 1:34 am PST) by jj

The shared web server ‘windhoek’ was submitted to an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We expect this process to take a several hours depending on the state of the file-system and we will provide frequent updates below as the check completes. However, we have already brought a new hardware online, and we’re serving your sites from backups, so you should be up and running. We apologize for the inconvenience that this situation causes!

We will keep this post updated with as much detail as we can provide. Please note, this is regarding the shared web server windhoek only, and no other servers or services (such as mail or mysql) have been affected. To check if you’re on this server, please check your user’s home machine in the DreamHost panel. If you have any questions or concerns, please contact support with the details, so we can assist you.

Update Mar 12th, 9:00pm PDT:  We should now have this issue resolved, all but two users are restored. We’ve emailed them directly. If you’re missing anything, please contact support — Brian H

Update Mar 12th, 7:00pm PDT: The restore continues on this server with no apparent issues. Services should stay up and running while the restore completes. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – Andrea

Update Mar 12th, 2:30pm PDT: The restore continues on this server with no issues that we’ve been able to find. Services should stay up and running while the restore completes. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – dan w

Update Mar 12th, 10:20am PDT: The restore continues with no current issues. Websites should be available and data restores are going as expected. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – Justin

Update Mar 12th, 4:55am PDT: The restore continues with no current issues. Websites should be available and data restores are going as expected. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – Chalise

Update Mar 11th, 1:45pm PDT: The restore continues with no current issues. Websites should be available and data restores are going as expected. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – Justin

Update Mar 11th, 6:25am PDT: This machine continues the restore process without any problems. Sites are still online and all apache processes are online. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – JJ G

Update Mar 10th, 11:50pm PST: We are showing no signs of trouble with this machine as the restore continues to execute! Sites and services continue to report onilne. Please feel free to contact support if you have any questions or experience any issues with your hosting on this server. – JJ G

Update Mar 10th, 5:43am PST: The load on this server is remaining under control for this restore and we are showing all services reporting online! Please feel free to contact support if you have any questions or experience any issues with your hosting on this server! -Patrick

Update Mar 10th, 1:03am PST: The load on this server is remaining under control for this restore and we are showing all services reporting online! Please feel free to contact support if you have any questions or experience any issues with your hosting on this server! -Patrick

Update Mar 9th, 7:56pm PST: This restore is moving forward without issues and all services are reporting online at this time. The server is stable and handling the load of the restore very well. We appreciate your patience while this process completes. Please contact support with any questions, or issues! -Dan J

Update Mar 9th, 2:30pm PST: This restore is moving forward without issues and all services are reporting online at this time. The server is stable and handling the load of the restore very well. We appreciate your patience while this process completes. Please contact support with any questions, or issues! -Oscar

Update Mar 9th, 4:15am PST: This restore is moving forward without issues and all services are reporting online at this time. The server is stable and handling the load of the restore very well. We appreciate your patience while this process completes. Please contact support with any questions, or issues! -Patrick

Update Mar 8th, 10:03pm PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Dan J

Update Mar 8th, 6:19pm PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Dan J

Update Mar 8th, 12:20pm PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Oscar

Update Mar 8th, 5:47am PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Patrick

Update Mar 8th, 3:19am PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Patrick

Update Mar 7th, 9.49pm PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! – Dan J

Update Mar 7th, 5:04pm PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! – Seohee

Update Mar 7th, 11:20am PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! – Oscar

Update Mar 7th, 5:20am PST: The restore on this server continues to run smoothly, without any issues. As we’ve mentioned previously, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! – Chalise

Update Mar 6th, 9:35pm PST: The restore on this server continues to run without issue. As mentioned before, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! -Dan J

Update Mar 6th, 4:15pm PST: The restore on this server continues to run without issue. As mentioned before, sites on this server might be a little slower than usual while the restore completes, but that should be resolved once it is finished. We thank you for your patience. Please contact support with any questions you have! – dan w

Update Mar 6th, 1:12pm PST: This server continues to be restored from backups and is currently running well! During a restore like this, the server may run slower than usual, but that will get better when this process is complete, so we appreciate your patience. Please contact support with any questions you have! -Brian S

Update Mar 6th, 9:40am PST: This server continues to be restored from backups and is currently running well! During a restore like this, the server may run slower than usual, but that will get better when this process is complete, so we appreciate your patience. Please contact support with any questions you have! -Justin

Update Mar 6th, 3:56am PST: This server continues to be restored from backups and is currently running well! During a restore like this, the server may run slower than usual, but that will get better when this process is complete, so we appreciate your patience. Please contact support with any questions you have! -Patrick

Update Mar 5th 7:33pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. If you’re having any issues with your sites,  then please contact support with the details so we can assist you. – Dan J

Update Mar 5th 4:00pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. The restore is continues as well. If you’re having any issues with your sites, it may be unrelated, so lease contact support with the details, so we can assist you. – Andrea

Update Mar 5th 12:00pm PST: We’re currently experiencing some connectivity issues that is causing services on this machine to be unavailable. This incident is unrelated to the recent issues this server has been experiencing on its own. Please keep an eye on this status post and we will update it as we have new information.

http://www.dreamhoststatus.com/2012/03/05/connectivity-issues-to-some-of-our-datacenters

Once that issue has been resolved and the service has been fully restored, we will return to updating this post with information specific to this machine. Please contact support if you have any questions or concerns regarding this. -dan w

Update Mar 5th, 9:50am PST: The restore from backup is continuing and services are up. All sites should be up and running. If you’re having issues, please contact support with the details, so we can assist. – Justin

Update Mar 4th, 8:30pm PST: The restore from backup is continuing, and without any issues. All sites should be up and running. If you’re having issues, please contact support with the details, so we can assist. – Andrea

Update Mar 4th, 4:00pm PST:  This server continues to mount files from the backup machine while we restore files locally. That process continues to run smoothly. The file system check continues to execute on the original hardware. If you have any problems loading your site please let support know. We will continue to post updates as this continues to execute. –dan w

Update Mar 4th, 6:20am PST:  Sites are still being served from backups and that process is stable. If you have any problems loading your site please let support know. As for the file system check, it continues to execute on the original hardware. We will continue to post updates as this continues to execute. –JJ G

Update Mar 4th, 4:45am PST:  The backups have been mounted to a new machine and are currently serving sites. In the meanwhile the file system check continues to execute on the original hardware. Further updates will continue to be posted, albeit a bit less frequently, as this issue progresses. –JJ G

Update Mar 4th, 4:10am PST:  Our admins informed us that due to the type of file system this check will take several hours. The above post has been edited to show this. An extra machine is being brought up with data from backups while the file system check executes. –JJ G

Update Mar 4th, 4:05am PST:  The check for bad blocks has been completed and the file system check is now in Pass 1, currently 3% complete. –JJ G

Update Mar 4th, 3:30am PST:  This file system check continues to execute and is currently checking for bad blocks, 87% complete through that pass. –JJ G

Update Mar 4th, 2:20am PST:  This file system check continues to execute and is currently checking for bad blocks, 45% complete through that pass. –JJ G

Update Mar 4th, 1:40am PST:  This file system check is proceeding rapidly and is currently checking for bad blocks, 11% complete at this point. –JJ G

This entry was posted on Sunday, March 4th, 2012 at 1:34 am and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

31 Responses to “Shared web server ‘windhoek’ moved to new hardware and restoring from backups (RESOLVED)”

Dears,

My website is still not working, and this problem is happening a lot these days don’t you think so?

Regards,

@Yazid Ajaleh: Your domain is not hosted on this server, so you aren’t affected by this problem. However I did look into your site problem and was able to get it back up. I’m sending you a follow up through email as well.

something wrong with ps55331, none of my sites there work! i can access the ftp but none is online.. been like that all day

@CJ: Your VPS is not affected by any server-wide outages at this time. Please submit a ticket with details about what problems you are having, because your VPS is up, the site you listed in your comment loads, and you haven’t written us about this downtime.

please need to know if my data will be recovered, it is imporatante also know when it will be ok

ARRRGHHHH!!!! THE TIMING OF THIS COULD NOT BE WORSE!!! ARGHHH!!! Thank you for your hard work in getting our sites back up, and I pray my two sites get back up before my 11pm deadline!!!!
– Lillie
http://www.AroundTheWorldL.com
http://www.TeachingTraveling.com

will there be compensation for this downtime?

Matt,

I second that. This outage has caused serious disruptions. I will tweet this question, too.

– Lillie

My sites has been down since noon’ish central time today. Are my sites on this server?

My Websites on this server are still really slow. Is this server almost fixed?

Lillie,

haha thanks for the extra support. I was hoping to finish a website today so I can collect payment for it. I understand that technical problems like this occur so I can’t be 100% upset about the downtime. I do however expect DH to help us out a bit. I don’t do e-commerce but image how much a site like that could be losing by the hour!

Can we get another update PLEASE. This is pretty amateurish.

@Julio – please note that you’re affected by a different outage, please check the updates there: http://www.dreamhoststatus.com/2012/03/02/shared-web-server-port-au-prince-moved-to-new-hardware-and-restoring-from-backups/
@Matt and Lillie – I placed a service credit on your accounts, apologies for the inconvenience.
@Crystal – you can check your user’s home server at https://panel.dreamhost.com/index.cgi?tree=users.users& – I checked for you, and you’re not on this server, and your site is loading fine. If you’re having any issues, please contact support through the DreamHost panel for assistance.
@Harrison – if you contact support through the DreamHost panel, we can check if your user is restored. I’m unable to check that without the specifics in your comment, unfortunately.
@John – Sorry about the delay, update is coming soon. We space out the updates once we have the server stabilized, but if you’re having any specific issues, please contact support through the DreamHost panel, so we can assist you.

It looks like nuuk is down, any updates?

First I’ve heard of this..Only noticed when my sites were taking too long to load. even the Live chat is offline.. Not good At least tell me future so I can prepare a contingency for this

@Peter – the server ‘nuuk’ is online, and has been for over 168 days. If you need assistance with your site, please contact support through the DreamHost panel with the details.
@Ryan – We’re working on better communicating incidents to our customers, but for now, please check dreamhoststatus.com in case your server is involved in a case. BTW, your domain that you linked to your comment is not on this server, and it loaded fine when I tried. If you need further assistance with your site, please contact support through the DreamHost panel with the details.

All my sites on vps ps55331 has down for over 24hours, and its over 18 since i picked a ticket.
No replies or respons on the ticket, i have had over 20 phone calls from angry customers..
Please take a look at this asap!
i can access the vps in ftp, but no sites appears in the browser..

You guys have disabled my account for over 6 hours without a) contacting me b) responding to my 8 extremely urgent tickets c) showing any concern for the fact that you’ve crippled my portfolio of 20+ websites on your overpriced VPS servers which in the past month has been down for hours on end!!!!

This is just getting ridiculous. If one of your techs is reading the support tickets and ignoring them this is completely unacceptable!

– Johann

@CJ: I am already looking at your issues.

@Johann: Your issues is not a support issue, our abuse team will get back to you soon.

Thiiiiiis stiiiinks. Still down, 24+ hours later. On the advice of a Dreamhost support person I switched from shared hosting to a VPS… now I’m freaked out because you VPS people are saying your sites are down, too! ARGH!
– L

@Lillie Marshall: I see a vps but there hasn’t been anything moved to it. We can move the data for you if you wish. The site is loading otherwise, but will be slower than usual until the restore completes.

I’m trying to go to panel.dreamhost.com to check which server my stuff is on, but I can’t reach that. Is there some other problem?

no panel. no e-mail, no website.

my site is loading sporadically and I just loaded a redesign for people to see. Please let me know what is going on.

I can’t get to the control panel to see what server this is on. Can you confirm it’s part of your announced issues, please?
Thank you!

My sites on Windhoek are still really really SLOW. Is this going to get better any time soon, or is it time to move to HostGator?

@John: Once the restore is more complete, it will be better. If you let me know a domain on your account, I can see what I can do to speed things up.

Oh the incompetence.

@Stacie: Your site is hosted on the shared web server “sugeknight”. There are no issues with it and your site is up without any issues.

There are still restore updates being posted daily. This server isn’t finished being restored still?

@John: Correct the server is still being restored, we will change the title to show resolved once the restore process finishes

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘castor’ moved to new hardware and restoring from backups (RESOLVED)
7:11 pm

Shared web server ‘castor’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 2nd, 2012 at 7:11 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘castor’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘castor’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 7th, 11:10am PST: This restore is now complete and all sites are up and resolving. If you notice any issues with your sites or are missing date, please contact support! – Oscar

Update Mar 7th, 5:13am PST: The restore process is getting very close to finishing! There are just a few users left to restore. All sites on this server should be up and running; if you notice any issues, please contact support! – Chalise

Update Mar 6th, 9:41pm PST: The restore from backups on this server is almost complete. There are only a handful of users left to restore. Sites continue to be served while this completes. We are showing no issues with the services on this machine, but please contact support with any questions or concerns you may have! -Dan J

Update Mar 6th, 1:15pm PST: The restore from backups on this server is almost complete. There are only a handful of users left to restore. Sites continue to be served while this completes. We are showing no issues with the services on this machine, but please contact support with any questions or concerns you may have! -dan w

Update Mar 6th, 1:15pm PST: This server is online and restoring from backups! We are showing no issues with the services on this machine, but please contact support with any questions or concerns you may have! -Brian S

Update Mar 6th, 10:15am PST: This server is online and restoring from backups! We are showing no issues with the services on this machine, but please contact support with any questions or concerns you may have! -Justin

Update Mar 6th, 4:27am PST: This server is online and restoring from backups! We are showing no issues with the services on this machine, but please contact support with any questions or concerns you may have! -Patrick

Update Mar 5th 7:33pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. If you’re having any issues with your sites,  then please contact support with the details so we can assist you. – Dan J

Update Mar 5th 4:00pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. The restore is continues as well. If you’re having any issues with your sites, it may be unrelated, so lease contact support with the details, so we can assist you. – Andrea

Update Mar 5th 12:00pm PST: We’re currently experiencing some connectivity issues that is causing services on this machine to be unavailable. This incident is unrelated to the recent issues this server has been experiencing on its own. Please keep an eye on this status post and we will update it as we have new information.

http://www.dreamhoststatus.com/2012/03/05/connectivity-issues-to-some-of-our-datacenters

Once that issue has been resolved and the service has been fully restored, we will return to updating this post with information specific to this machine. Please contact support if you have any questions or concerns regarding this. -dan w

Update Mar 5th, 9:50am PST: The restore from backup is continuing, and without any issues. All sites should be up and running. If you’re having issues, please contact support with the details, so we can assist. – Justin

Update Mar 4th, 8:32pm PST: The restore from backup is continuing, and without any issues. All sites should be up and running. If you’re having issues, please contact support with the details, so we can assist. – Andrea

Update Mar 4th, 4:00pm PST:  The restore process continues without issue and the machine remains stable. If you do see any problems with your site, please contact support. Thank you for your patience while we get this server back to normal. –dan w

Update Mar 4th, 6:20am PST:  The restore process is still running and the machine remains stable. If you do see any problems with your site, please contact support. Thank you for your patience with this issue. –JJ G

Update Mar 4th, 12:52am PST:  The restore process continues smoothly and, again, we have no new issues to report. If you do see any problems, please contact support. –JJ G

Update Mar 3rd, 7:51pm PST:  The restore process is still humming along, so no new issues to report. Again, if you’re seeing problems, please contact support –Brian H

Update Mar 3rd, 12:30pm PST: The restore process is running without any issues. Apache services on this host are online and responding so all sites should be up. Please contact support if you have any questions or if you are experiencing any issues! Thank you for your patience. -Oscar

Update Mar 3rd, 7:00am PST: The restore process continues to execute and no issues are being reported. Apache services on this host are online and responding. Please contact support if you have any questions or if you are experiencing any issues! Thank you for your patience. -JJ G

Update Mar 3rd, 2:02am PST: This restore is progressing along with no issues being reported. All Apaches on this server are online and things are looking good. Please contact support if you have any questions, or if you are experiencing any issues! -Patrick

Update Mar 2nd, 10:16pm PST: The new hardware is up and all apache instances are resolving. Backups have been mounted and the restore is running. At this time all sites are running from mounted backups. Please contact support with any questions! -Dan J

This entry was posted on Friday, March 2nd, 2012 at 7:11 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

61 Responses to “Shared web server ‘castor’ moved to new hardware and restoring from backups (RESOLVED)”

The control panel is down … cannot access. Could you check?

Panel is down!

Control Panel is down for me as well

BTW Does this happen often (panel down) – curious as I am new to Dreamhost

IQ not really … as any hosting they have several hardware problems but you can always access the control panel in general

Database and control panel both are down for me.

By the way can someone from support confirm you received the message that the control panel is down? Could you let us know when it will be back?

Can somebody please help??? There are lots of issues. Mine is VPS even then it is affected by a shared server hardware change?

Thank Diego – How often does it go down in the year. Also how is the bandwidth and general services of Dreamhost.

IQ- Their services had been not so bad previously but this month has been crazy. This is third time we are out of services. I am not sure what is happening. How can a web hosting company’s servers can be so fragile?

Thanks Ritu

Since I am new to Dreamhost this could be a good time to Exit – any suggestions on a reliable and professional webhost ???

IQ

Been with them for several years and they have been the best I have dealt with (use to switch every 3-6 months previously)

Every webhost you go with will have down time.

How they deal with it is what you want to know.

Dreamhost has typically been very good so while the past month may have had issues, the fact that they do fix things and try to work out why something is wrong is better than most I have dealt with.

Last time they had issues like this, once they sorted them out they were good for a while so I expect current issues will be sorted soon enough.

Time will tell.

panel is now working for me

IQ – I am not so sure. Its your call but considering that the next few weeks are very crucial for us, I am loosing confidence.

Panel is working but the database is still down. I am trying to reboot from panel but it is giving the status as failed.

Does anyone know of any problems with the server “tarawa-atoll”? I haven’t been able to access my sites due to the following error.

Warning: require(/home/username/website.com/wp-includes/post-template.php) [function.require]: failed to open stream: Permission denied in /home/username/website.com/wp-settings.php on line 122

Fatal error: require() [function.require]: Failed opening required ‘/home/username/website.com/wp-includes/post-template.php’ (include_path=’.:/usr/local/lib/php:/usr/local/php5/lib/pear’) in /home/username/website.com/wp-settings.php on line 122

Hello there!!! Any updates on when the MySql server will be back up? Please Please Please Help!!!

Please update….

I understand that you are busy in solving the problems and trying your best but we will really appreciate if you could update us with the status. Our services are down for a long time now.

@Diego, Ritu, and IQ: We did have a panel outage this morning. Our admins were able to get that resolved and it’s running well.

@Derek: Unfortunately I’m not able to find an account with the email address you used on the comment. Please contact support and let us know the domain with the issue, we’ll be more then happy to help.

https://panel.dreamhost.com/index.cgi?tree=support.msg&

@jj @DanJ
Can you please check out ps37457? My VPS has been down for almost a week now. I can back up for a day with the wrong files, I have sent a couple of support tickets with no reply. Can you please check out this VPS and help me get it back online? PLEASE!

Hi JJ – The panel was working fine – Thanks.

I am asking about our database server. Can you please update on that?

And I have raised tickets in Support but there is no reply. What do I do now?

I am a customer since 06, I just decided to pay for a support phone call back in order to get to the bottom of this. I have been overly patient, how can you have over a week of downtime and almost zero communication with your customers. Looking forward to your call back!

And I have paid for the support phone call back thrice. The only time I get a call back is when the services are up. What is the use of paying for a call when the services are already up? I am sorry if its sounds rude but we are in crucial moments and we need our services 24×7. This is the reason we upgraded to VPS but I am not sure whats happening lately.

Can somebody please update me if you are looking into my database problem?

@IQ, check out my posts on the other page, about port-au-prince being moved, to learn why you are right to escape from this horrendous company while you can…

I dont want to be a spammer but support is neither informing me or as it seems hundreds of other clients the normal way, so I dont consider this to be spam either as I am left with no other option.

My websites have been down for 24h now, clients are raging and thank God it is weekend or I would have been a dead man walking by now although my biggest client have phoned me and said they started a one week TV campaign which shows an ad ending with leading people to their new fantastic website (saw it five minutes ago on TV). I have been glued to the computer, refreshing this page and my support panel since I woke up this morning, for 10 hours now.

This is a web company owners worst nightmare.

Not only have the complete lack of feedback combined with growing amount of phonecalls slowly turned me into a nervous wreck but when I started looking at what other customers have written on the update pages here for other servers than mine, which have been through the same operation lately, I realize that

a) many of these processes have taken a WHOLE WEEK or more

b) many of the customers have NOT recovered their initial information

I mean, check out this response to one comment here:

“Hi Daniel,
I apologize, I’ve gone through all of our backup servers, and have been
unable to find backups for any of these users. While we do our best to
maintain backups for all users, we can’t guarantee their availability,
and you’ll need to restore these from your own backups. I’m very sorry
that there’s nothing more we can do for you in this matter.
Thanks,
Brandon”

I have no idea what to do, I dont know what to tell clients, I cannot do any work because I have no idea what my work schedule will be like for the next hours or days or weeks, my entire company and current life situation seem to be in the hands of Dreamhost and based on my experiences during the last 24 hours and what I see others writing here, it seems more likely that I will have a heart attack than my websites being back online before monday morning…

If my websites arent back online by tomorrow I will dedicate my entire day to E-mail every newspaper in the world to tell them about one of the biggest hosting companies online having thousands of business websites offline without proper explanation or follow up, for days. On these pages there should be plenty to write about…

Hopefully that will either cause Dreamhost to take this seriously and/or go bankrupt.

Haaaaaaa haaaaaaa…. free dreamhost..

@Insanity: What is your webid or domain name? The link to your name is not hosted with us. The comment you mentioned was resolved and ended with Daniel thanking me personally since I resolved the issue for him. We are definitely responding to tickets and if you can let me know your domain name hosted here or webid, I can respond to you as well.

@Ritu: Your sites are loading for me without any issues. Which sites aren’t working for you? Let me know.

@Oscar, I feel no need to put my customer id at the moment as I simply dont trust you guys, I am an extremely unhappy customer and I am suffering big economical losses from this. In none of my posts have I mentioned other hosting companies so I take for granted that people see my views as credible.

Also, I find it exceptionally amusing and also hipocrite and false advertising on your behalf that I have been trying anything in the world to get feedback from you guys via support, which you yourself has stated here numerous times, with no response whatsoever and here you come playing servicefriendly in front of everyone on a comment page which clearly states “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost”

I mean, seriously, isnt it the epitomy of bad follow up when you respond me on a comment page which you dont use for responding, while you fail to respond for days in my support panel?

:-)

@Insanity: We can’t help you out or check what’s going on with your sites if you don’t let us know how to check your account. If you let me know one domain you host here that will be enough.

I also sent a ticket for Database issues (thus many of my domains aren’t loading) and it’s been 24 hours now of full down time and no response via my support ticket.

Maybe Oscar can help. :)

So, all our websites went down and Dreamhost claim it was unexpected and left them no chance to notify us first… Make up your mind yourselves, below I have posted all REPORTED MAJOR OUTINGS for the last 10 weeks – notified downtime (17 in this period) and other downtime which isnt notified on dreamhoststatus.com is not included.

There are 108 in total, compared to 17 notified ones, making rougly 10 major outings a week! Now, what we need to ask ourselves is not only if DH’s claims that the outings of Castor and Port-au-prince the last two days can possibly be called “unexpected” or if Dreamhost should be allowed to falsely sign up new clients using a “100% uptime guarantee” statement on their website, but whether they are worth all this bother at all.

Downtime Dates & headlines of major outings (only) from dreamhoststauts.com

14.12.2011 – Web VPS host ‘homie-vserver81′ offline for hardware issues
14.12.2011 – Shared web server baham offline for RAID card replacement
14.12.2011 – Web VPS host ‘homie-vserver24′ being restored from backups on new hardware
14.12.2011 – Web VPS hosts homie-vserver265 and homie-vserver286 offline due to network issues
14.12.2011 – Network issues in Irvine datacenter causing delays to server updates
14.12.2011 – Shared web server ‘sainthelier’ offline for unscheduled file-system check
15.12.2011 – Shared web server ‘astana’ back online on old hardware
15.12.2011 – The minkar server is under DDoS attack
16.12.2011 – Shared web server ‘warriors’ offline intermittently due to Denial of Service Attack
17.12.2011 – Shared web server ‘johnson’ restoring from backups on new hardware
17.12.2011 – Shared web server ‘hagatna’ down for system file check
17.12.2011 – Shared web server ‘floyd’ being moved to new hardware
19.12.2011 – Shared web server ‘astana’ restoring from backups on new hardware
20.12.2011 – WebFTP temporarily offline
20.12.2011 – Web VPS host ‘homie-vserver147′ down due to hardware issues
20.12.2011 – Web server ‘porto-novo’ offline intermittently due to denial of service attack
21.12.2011 – Web server ‘porto-novo’ offline intermittently due to denial of service attack
25.12.2011 – Shared web server ‘tech21’ restored from backups on new hardware
25.12.2011 – Shared web server ‘minkar’ was under DDoS attack, one apache getting new IP
26.12.2011 – Networking issues affecting a row of machines at our LAX datacenter
28.12.2011 – Irvine datacenter networking issues
29.12.2011 – Intermittent downtime on shared machine ‘Wellington’ due to RAID issues
29.12.2011 – Panel down
29.12.2011 – ‘Manage Account’ area of Panel down
29.12.2011 – Shared web server ‘betria’ offline due to unscheduled RAID card
30.12.2011 – Shared web server ‘alcyone’ restoring from backups on new hardware
30.12.2011 – Web VPS host homie-vserver175 offline for unscheduled file-system check
30.12.2011 – DreamHost Panel offline
31.12.2011 – Web VPS host homie-vserver140 offline for unscheduled file-system check
02.01.2012 – Shared web server “crater” moving to new hardware
02.01.2012 – Messages originating from Webservers are being delayed
02.01.2012 – WebMail ( Squirrel Mail ) minor outage
03.01.2012 – Ongoing networking outage in one of our facilities
03.01.2012 – Email Relaying Delay
04.01.2012 – Webserver VPS “homie-vserver150? offline for unscheduled RAID card replacement
04.01.2012 – Webmail issues being investigated
04.01.2012 – Homiemail-sub5 mail service expansion
05.01.2012 – Web VPS host homie-vserver174 is being moved to new hardware
05.01.2012 – Shared web server ‘miaplacidus’ down
06.01.2012 – web server “tabbuts” is currently down.
07.01.2012 – Shared web server ‘mcgurn’ down for unscheduled file system check
07.01.2012 – Web VPS host homie-vserver190 being moved to new hardware
09.01.2012 – Shared machine ‘ Godin ‘ is moving to new hardware
09.01.2012 – Panel down
10.01.2012 – WebFTP outage
10.01.2012 – Shared web server ‘manama’ being moved to new hardware
11.01.2012 – Shared mail service issues on homiemail-sub5 is being investigated
11.01.2012 – Shared web server ‘hachiman’ offline due to unscheduled RAID card replacement
12.01.2012 – Shared web server “yangon” down due to hardware issues
13.01.2012 – Shared web server ‘dodoma’ offline for RAID card replacement
13.01.2012 – Network Outage affecting one of our Data Centers
13.01.2012 – Intermittent Stability Issues on WebMail
14.01.2012 – Shared web server ‘porto-novo’ under DDoS attack
15.01.2012 – Shared web server ‘tygra’ moved to new hardware and restoring from backups
17.01.2012 – Shared web server ‘dobro’ moved to new hardware and restoring from backups
17.01.2012 – Web VPS host ‘homie-vserver154′ moved to new hardware and restoring from backups
18.01.2012 – Shared web server ‘samick’ offline for RAID card replacement
18.01.2012 – Intermittent Webmail Stability Issues
18.01.2012 – Emergency Maintenance with our Easy one-click installs
18.01.2012 – Web VPS host ‘homie-vserver134′ moved to new hardware and restoring from backups
18.01.2012 – Major network outage
18.01.2012 – Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups
18.01.2012 – Emergency network maintenance
19.01.2012 – Hambone MySQL Downtime Due to RAID Issue
19.01.2012 – Intermittent Mail Instability on homiemail-sub5
20.01.2012 – Shared web server ‘portvila’ moved to new hardware
20.01.2012 – Changing Shell/FTP Passwords due to Security Issue
21.01.2012 – Web VPS host homie-vserver198 moved to new hardware and restoring from backups
21.01.2012 – Password changes delayed in the web panel
22.01.2012 – Web VPS host homie-vserver233 moved to new hardware and restoring from backups
22.01.2012 – Shared web server ‘albali’ under file system check and being moved to new hardware
22.01.2012 – Degraded performance on personal backup server hanjin
23.01.2012 – Shared web server ‘torr’ moved to new hardware and restoring from backups
26.01.2012 – Connectivity issues to a small subset of web servers
29.01.2012 – Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down
29.01.2012 – Webmail login issues
30.01.2012 – Easy/Simple One Click Installs temporarily offline
30.01.2012 – Connectivity issues in one of our datacenters
30.01.2012 – Shared web server Sinistra having issues
02.02.2012 – Email connectivity issue in one of our datacenters
02.02.2012 – Shared web server ‘portvila’ not accessible
03.02.2012 – Web VPS host homie-vserver168 being moved to new hardware and restoring from backups
03.02.2012 – Web VPS host homie-vserver163 being moved to new hardware and restoring from backups
04.02.2012 – Connectivity issues with our Irvine Datacenter
07.02.2012 – Mail bouncing for subset of customers
09.02.2012 – Web VPS host homie-vserver206 being moved to new hardware and restoring from backups
13.02.2012 – Web VPS host homie-vserver158 being moved to new hardware and restoring from backups
14.02.2012 – Mailservers on Trend Micro RBL, working on removal from list
17.02.2012 – Web VPS host homie-vserver197 restoring from backups to new hardware
17.02.2012 – Web VPS host homie-vserver173 offline for file-system check
17.02.2012 – Shared web server ‘paramaribo’ offline for RAID card replacement
18.02.2012 – Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware
21.02.2012 – Shared web server ‘cayenne’ restoring from backups to new hardware
22.02.2012 – Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware
22.02.2012 – Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups
23.02.2012 – Web VPS host ‘homie-vserver131′ back up and running on original hardware
23.02.2012 – Shared web server ‘stampeders’ restoring from backups
23.02.2012 – Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups
24.02.2012 – Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups
25.02.2012 – Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups
27.02.2012 – Web VPS host homie-vserver142 moving to new hardware and restoring from backups
27.02.2012 – Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups
28.02.2012 – Shared web server ‘mystique’ moved to new hardware and restoring from backups
29.02.2012 – VPS host machine ‘homie-vserver288′ under DDoS Attack
29.02.2012 – Connectivity issues in one of our datacenters
01.03.2012 – Shared web server ‘singapore’ moved to new hardware and restoring from backups
02.03.2012 – Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups
03.03.2012 – Shared web server ‘castor’ moved to new hardware and restoring from backups

Oh, and dont forget, Dreamhost claim the port-au-prince and castor outings were U N E X P E C T E D :-)

Why is Dreamhost deleting comments about downtime history others are posting?????

Yep, I have had two posts deleted yesterday and today, the last one showed a complete list of all 108 unexpected outings since 14/12-11.

I dont know why they deleted it, when anyone can view it themselves by simply going backwards on dreamhoststatus.com and see it for themselves. Maybe the truth hurts? Or maybe it was because I stated that their argument that the outages this weekend of port-au-prince and Castor were UNEXPECTED must be considered a blatant lie to their own customers, bearing in mind that they for months have seen their servers one by one having had to perform the same operation?

But, as I said the last deleted post showed that the last 10 weeks, Dreamhost have had – in addition to all the other downtime which isnt recognized – a total of 108 major outings, which is about 11 a week, more than one a day, compared to 17 notified outings.

The other comment from yesterday which also was deleted was when I addressed the honesty subject and asked Oscar if he really thinks his clients are so stupid that they think these things happen unexpected. I mean, isnt it strange, people, isnt it almightly LUCKY on their behalf that all these UNEXPECTED outings are spread over months with more or less one every day?

What a weird coincidence, eh? Incredibly lucky I must say, that while almost every server is subject to such an UNEXPECTED incident, it is nicely spread over a three month period??? Wow, thats weird, that for example five of them dont break down at the same time but instead spread themselves over a ten day period. Because if this is a coincidence, we should all be scared shi..less, because it means that the whole Dreamhost thing could blow any time, and if it is no coincidence it is even worse because that would mean that our jobs and everyday life as we know it is in the hands of liars and cheats.

As I stated before, my posts are valid because I havent mentioned other providers or anything, I am only copying and pasting true stats from this very page, and i am only addressing something we all know. It is already clear that some of us have lost entire websites, maybe weeks or months of work, and there is no way Dreamhost will compensate for that. So all I ask for on everyones behalf is a bit of honesty.

We, who have already fallen, are pretty much sc..wed as it is too late for us – we have to accept either we want to or not, that most of our websites have been reset a month back in time while the ones we created lately are completely lost – we can do NOTHING about it. However, I do demand onbehalf of all the others who dont visit this website and comment page, who right now are sat in their homes or offices sweating away creating websites, that they are informed about the problems Dreamhost are having!!!!

I have to accept today that I have lost lots of work (= money) without compensation, and while it doesnt make it acceptable, it would have made a world of difference to me if Dreamhost, especially as I since Xmas have sent numerous requests, error messages and complaints, if they had TOLD me that they are having such problems that their entire system was in jeopardy. Why didnt they send me a mail advising me to do regular security copies as the system was unusually instable? Yeah, it would have concerned me big time but they would have saved themselves the lawsuit im launching now.

So, I beg you Oscar and the rest of you, NOT TO DELETE THIS POST because it makes you all just look worse AND PLEASE PLEASE notify other users who have no idea that these days all their websites will disappear for days and then either get reset to a month back or lost forever!!!!!

Yep, I have had two posts deleted yesterday and today, the last one showed a complete list of all 108 unexpected outings since 14/12-11.

I dont know why they deleted it, when anyone can view it themselves by simply going backwards on dreamhoststatus.com and see it for themselves. Maybe the truth hurts? Or maybe it was because I stated that their argument that the outages this weekend of port-au-prince and Castor were UNEXPECTED must be considered a blatant lie to their own customers, bearing in mind that they for months have seen their servers one by one having had to perform the same operation?

But, as I said the last deleted post showed that the last 10 weeks, Dreamhost have had – in addition to all the other downtime which isnt recognized – a total of 108 major outings, which is about 11 a week, more than one a day, compared to 17 notified outings.

The other comment from yesterday which also was deleted was when I addressed the honesty subject and asked Oscar if he really thinks his clients are so stupid that they think these things happen unexpected. I mean, isnt it strange, people, isnt it almightly LUCKY on their behalf that all these UNEXPECTED outings are spread over months with more or less one every day?

What a weird coincidence, eh? Incredibly lucky I must say, that while almost every server is subject to such an UNEXPECTED incident, it is nicely spread over a three month period??? Wow, thats weird, that for example five of them dont break down at the same time but instead spread themselves over a ten day period. Because if this is a coincidence, we should all be scared shi..less, because it means that the whole Dreamhost thing could blow any time, and if it is no coincidence it is even worse because that would mean that our jobs and everyday life as we know it is in the hands of liars and cheats.

As I stated before, my posts are valid because I havent mentioned other providers or anything, I am only copying and pasting true stats from this very page, and i am only addressing something we all know. It is already clear that some of us have lost entire websites, maybe weeks or months of work, and there is no way Dreamhost will compensate for that. So all I ask for on everyones behalf is a bit of honesty.

We, who have already fallen, are pretty much sc..wed as it is too late for us – we have to accept either we want to or not, that most of our websites have been reset a month back in time while the ones we created lately are completely lost – we can do NOTHING about it. However, I do demand onbehalf of all the others who dont visit this website and comment page, who right now are sat in their homes or offices sweating away creating websites, that they are informed about the problems Dreamhost are having!!!!

I have to accept today that I have lost lots of work (= money) without compensation, and while it doesnt make it acceptable, it would have made a world of difference to me if Dreamhost, especially as I since Xmas have sent numerous requests, error messages and complaints, if they had TOLD me that they are having such problems that their entire system was in jeopardy. Why didnt they send me a mail advising me to do regular security copies as the system was unusually instable? Yeah, it would have concerned me big time but they would have saved themselves the lawsuit im launching now.

So, I beg you Oscar and the rest of you, NOT TO DELETE THIS POST because it makes you all just look worse AND PLEASE PLEASE notify other users who have no idea that these days all their websites will disappear for days and then either get reset to a month back or lost forever!!!!!

and, im “insanity”, after the last post Dreamhost BLOCKED me from posting here more….

so that shows you their priorities and level of honesty, no? Blocking customers who are unsatisifed and tell the truth…

@I was deleted and blocked & @Frank: Your posts are very spam like and the wordpress spam filter had moderated them. I approved them so they show again, but must ask you again to not abuse the comments of the status blog in repeatedly posting long comments. I would suggest contacting us for support or for the issues you are having, we cannot and will not address your specific issues here.

@Sean: Your sites are currently loading and I replied to your ticket.

Im sure Dreamhost can appreciate the facts that these people post these things because they are unhappy, confused or even in complete panic over their websites being down for 4-5 days without prior notification?

Also, everyone can see that they, just like myself, have been over and oevr again trying to contact you via supprt with no response?

i mean, if you guys constantly are going to respond to all critics by asking people to address things via support, tehn at least answer them when they do so, right?

@Tomas: We do understand and apologize for the required moved to new hardware. There was absolutely no way to give prior notification, the raid failed, and that was when the move to new hardware had to start.

There is currently only 1 open ticket for the castor server. Other servers may have higher support volumes and there could be longer response times, but we are doing our best to catch up and fix all issues.

@Justin,

Question 1: Is this a common thing which happens to your servers all year round, I mean is this part of being registered on Dreamhost? Are you currently having more problems with your hosting than usual or do you consider things to be as normal? Please answer this.

Question 2: You guys have as we can see stated that it was unexpected, several people here have dwelled on this and I cant find any attempt to respond to this from you so I drag it on then – there are two factors which makes this statement look very hard to believe:

– If these outages, referring especially to the ones where you each time state that an “unexpected problem has caused raid blablabla” where people lose their work and websites if they havent backed up recently, if these outages really ARE unexpected and coincidental, how do you explain that they are nicely spread so that there is one every 1-3 days throughout the year? Are you guys just extremely lucky, is God being good to you etc, how do you explain this? Can you understand that it is difficult for us to believe that you dont know about these outages caused by change of hardware and actually are the ones who spread them over time and in a workable order yourselves? The possibility that these unexpected outages spread systematically over time and never ever crash for example five or ten at a time (especially after Brendan stated you have 1000+ servers) seems extremey unlikely…

– Regardless of whether you know about them beforehand or not, I have seen now here that you had over 100 outages this year or something, lots of them being exactly the same raid thing which we experience here. Then, how can that be regarded as unexpected? Surely it is not unexpected if a server goes down with the same problem every 2 days? If your response to my question 1 is that this is normal all yea round, then there should be a clear message in the panel to remind people to back up their site almost daily due to hosting life being like this PLUS you shouldnt state as sales argument on your mainpage that you guarantee 100% uptime. If this is NOT normal, then you should inform everyone about this being an unusual problem at this moment and urge everyone to back up often.

Please answer these questions? I just cant figure out why you state that it was unexpected and why you dont inform people beforehand. Why would you every 1-3 days deliberately let your customers be surprised by a “server being moved to new hardware due to raid problem” error and lose weeks or months of work, entire websites and even clients? I just done get it, and im sure mst people dont either? Statistically, we will within a day or two see another shared or VPS server go down with a message here which says the same, people posting desperate messages and you stating that it was unexpected.

Surely this cant remain this way??? Will you change your routines?

The question is in general, not this specific server.

@Thomas – I’ll try and address your questions. This hardware move was indeed unexpected – according to the machine history, the server hasn’t been rebooted or crashed since December, so in fact, this was an unexpected failure. As soon as we recognized that it wasn’t a simple reboot situation, we initiated the restore to new hardware from backups. As you see, there was nothing indicating that the server would fail and hence the lack of notifications. I would like to assure you that we do not deliberately bring downtime to our customers. One could argue about what ‘normal’ is in hosting service. All I can tell you is that our readiness for unexpected incidents is being improved, but it doesn’t take away from the fact that there is and there always will be hardware failure – this is part of *any* hosting situation. Also, one machine failing doesn’t guarantee that another will or another won’t. Our guarantee is also 100% uptime, which means we compensate you if it isn’t, based on what’s posted at http://dreamhost.com/web-hosting/. If you have any questions, please let me know.

@andrea,

Sure I got questions, and thank you for your response;

1 – You say you compensate people who havent got 100% uptime. I assume that you then have a system in place where you in an organized way inform AUTOMATICALLY deduct an amount from the fee of EVERY user on each server which goes down or at least send them a message? Or is this a privilege which only is reserved for those who demand it themselves? When I look back your history, almost a hundred servers have failed only the last few months – please confirm here and in front of everyone whether ALL users on these servers received this compensation or only those who demanded it. Or just a few?

2 – What do you mean with compensation? How do you calculate it, please inform us about this in detail? In my case for example, my hundreds of websites have been down for 3 and 4 days, while I have completely lost 11 websites and one web portal. It is only 10.00 monday morning and I have received two messages from clients already who demand their money back. I estimate this having cost me around TEN THOUSAND DOLLARS and I will have to use today and tomorrow entirely to save face in general and avoid being sued. Meanwhile, you have notified me via support that you compensate for TEN DOLLARS, not ten thousand dollars, and I still suspect I have achieved more than most. How do you calculate this? Do you compensate people one dollar for 3 days lost? Please explain.

3 – You state again, like on your website, that you GUARANTEE 100% uptime. How do you define the word guarantee in this case? I guarantee my my staff the job I contracted them to, I cant just tell them every now and then that this week they are without a job and income as I guaranteed they would. I contract websites and ask clients to pay me money for them, while in change for these money I guarantee things. I guarantee my grilfriend I will not sleep with others. I guarantee society that I will not punch people. In your case, though, you even today continue to GUARANTEE 100% uptime on a page where everyone who reads it has had their websites offline for days – please explain to us how Dreamhost define this guarantee which we all know is not a guarantee in the sense that WE know this word?

writing here following Tomas suggestion on port-au-prince discussion.

@Tomas you are right man. I’ve lost 20 days of work on my sites and fully lost an user with all his data. A taunting 10$ compensation is not the solution I think.

Another point is that you state you GUARANTEE 100% uptime… Ok where can I find the SLA whith relative compensation details?

I just received a support message telling me any work on my hundreds of websites done since 6th of january is lost…

HAPPY HOSTING is their slogan, and they GUARANTEE 100% uptime

Shouldnt be DREAMhost, but NIGHTMAREhost!

We lost an important project … last (dreamhost) backup seems to be 6th January.

I’m losing faith and hope… two month work just vanished and probably i’ll be sued by the customer (which is right by the way).

@Tomas: 1) No, please read the uptime guarantee. It is like any other SLA, you need to write in to receive it. 2) This is detailed in the uptime guarantee, limit of liability in the terms of service. 3) The guarantee is defined in the text on the website.

@matteoparrucci: http://dreamhost.com/web-hosting/ Satisfaction Guaranteed tab at the bottom.

@Filippo: You are not on castor. The issue with your site is specifically related to port-au-prince, castor does not have this issue and has full backups and repaired old hardware.

Yet another site of mine got automagically deleted by dreamhost. I have spent two years and thousands of dollars developing a superb website for a world famous opera singer. Now I have only a favicon to show for it; that’s all that’s left of it. Thanks dreamhost.

@justin,

Ok, thank you, basically you have confirmed that the 100% guaranteed uptime is a worthless bag of potatoes which the terms and condition twists leave you completely unliable for no matter if its up or not

@Sadim: There is not enough info in your post to tell you anything. Please contact support with specific details if you would like assistance.

@Tomas: I would encourage you to check the terms of service for any other host (or any software product or similar services), these are standard legal methods that all webhosts use.

@justin,

are you saying “it is standard legal methods all webhosts use” to state on their mainpage a sales promise of “100% uptime guarantee” and then have one or more servers down every day?

I have asked you, on behalf of everyone, to explain in normal words what the 100% uptime guarantee means, if it doesnt mean 100% uptime???

Is it so hard for you, dont you have a more informative and customer friendly response than “read the terms and conditions”?

Just tell us all why you paint the “100% uptime guarantee” all over the place when you dont have 100% uptime!

I mean, why dont you advertise FREE HOSTING or 1000 TB RAM then, if it doesnt matter whether you actually provide it or not? EXPLAIN!!!

I too am sick of downtime, but really: posting a list of blog entries from dreamhoststatus is not an argument, that’s over _all_ dreamhost servers. What matters is looking at a single server (or, taking an average of single-server downtime). Think about it: say that the worldwide chance of a server being down a given day (across all hosters) was 1 %, if Dreamhost had 100 servers, then any given day, averaging, chances are one will be down. The more servers a hoster has, the bigger the chances of any one of them being down, simple statistics. So if you want to make an empirical argument, give a list of downtime for a single server, or averaged across single servers.

no e-mail, no panel, and forget the website … haven’t seen it reliably in 7 days!

good point, Kevin, I agree

Dreamhost, do or dare? Will you post the stats for the uptime of this particular server for the last 12 months?

:-)

Probably they could do it… But the funny thing is that they would not consider loosing more than 2 motnhs of work (6jan -> 6mar and going on) downtime because the site is running.
And if you have created users and they are lost? And if you modified files and can’t get them back?

The real problem I see is about backups policy and reliability. Which is the backup policy? Explain us. Once every year and we was lucky all this happened in march and not in december?

The backups lives in the same raid of the original data (useless backups)

@Tomas: We can provide a customer with the reboot history of their server if requested, but “uptime” could mean a lot to different people. To us, uptime is that the webserver and apaches are running. To a customer, uptime could mean that plus their site is loading properly, which is something that we can’t monitor for every individual 1+ million sites. For example, my 6 year old employee account is on hawkman and I use it for testing. The last year for hawkman reboots includes (2011-01-20 03:42:40 reboot, 2011-06-13 10:17:20 reboot, 2011-09-16 00:10:23 reboot). The last reboot was for a firmware upgrade for the raid controller. A given webserver doesn’t get dozens of reboots a year, but ones restoring from backups may have issues and a higher number of reboots until it completes.

@matteoparrucci: Customers are entirely responsible for their data and backups, as per the terms of service. Our backups are done for the sole purpose of recovering from unforeseen hardware failures, and they are provided as a convenience to to customers via the restore buttons for database and websites. This is pretty common with all web hosts. The backups are done to separate large storage servers, the original raid would not affect the backups.

@Justin You are still not responding to the original question:

Which is the backup policy? Explain us. Once every year and we was lucky all this happened in march and not in december?

As you said “Our backups are done for the sole purpose of recovering from unforeseen hardware failures”. I think this is the case of “unforeseen hardware failures” and give back data 2 months old can NOT be called recover. Recover could be restore 3 days or 1 week old backups.

That said I’ve found another problem on my account. I had a backup folder containing backup scripts and last year backupped data. I yearly clean this folder and move my backups to another storage. All the content of this folder disappeared and it was created long time before 6th jan. The cronjob that calling my backup script also vanished. Do I have to manually check all my account for other missing data?!?!?!?

@matteoparrucci: There is not a backup policy about how old backups may be. We want them to be newer, so we are working on adding new hardware, monitoring and processes. Currently some servers have backups that were only a week or so old, some like yours unfortunately are in need of improvement.

As for the backup folder missing, we do not allow storing of backups on our shared servers (see unlimited policy), so they should not exist there in the first place. Our backup processes will also ignore them (see a quick note on backups of backups).

lol what a idiot policy -_- if I name a project bkup because I like B cup girls it will not be backupped. Good to know I’ll name my backup scripts folder “donotremovemeplease” so it will be keeped regardless raid failures if you have a policy for this name too -_-

btw this folder contained the backup script and last week backups that was not intended to be stored but to be moved to other (safest) storage every few weeks. Not trusting you worked in the end.

@matteoparrucci: Sorry if you are not satisfied with our backup setup. Thanks for your feedback, we will consider it for future changes.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups (RESOLVED)
5:55 am

Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 2nd, 2012 at 5:55 am PST) by patrickm

Due to issues with the RAID controller on the shared web server ‘port-au-prince’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘port-au-prince’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 7th, 4:54pm PST: The restore is now complete! As mentioned in previous updates, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. Thanks for your patience! – Seohee

Update Mar 7th, 11:20am PST: The restore is continuing to run smoothly, though the load is slightly elevated due to the restore process which may cause your sites to run slowly; this will clear up once the restore is complete. As mentioned in previous updates, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. – Oscar

Update Mar 7th, 5:09am PST: The restore is continuing to run smoothly, though the load is slightly elevated due to the restore process which may cause your sites to run slowly; this will clear up once the restore is complete. As mentioned in previous updates, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. – Chalise

Update Mar 6th, 9:43pm PST: The restore is still moving along on this server. As mentioned before, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. -Dan J

Update Mar 6th, 4:24pm PST: The restore is still moving along on this server. As mentioned before, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. -dan w

Update Mar 6th, 1:24pm PST: This restore is continuing. The load is still a bit higher than normal and the sites a bit slower, but this will return to to normal when the restore completes. We are using a backup from Jan 6th for this restore, so any data uploaded since that time will need to be re-uploaded by you now. If you have any questions or issues with your hosting, please contact support for assistance! -Brian S

Update Mar 6th, 10:15am PST: This restore is continuing. The load is still a bit higher than normal and the sites a bit slower, but this will return to to normal when the restore completes. We are using a backup from Jan 6th for this restore, so any data uploaded since that time will need to be re-uploaded by you now. If you have any questions or issues with your hosting, please contact support for assistance! -Justin

Update Mar 6th, 4:54am PST: This restore continues to progress! We are using a backup from Jan 6th for this restore, so any data uploaded since that time will need to be re-uploaded by you now. If you have any questions or issues with your hosting, please contact support for assistance! -Patrick

Update Mar 5th 8:33pm PST: The restore process continues, we appreciate your patience. If you are seeing any problems with your sites please contact support. Please note, the backups are from 1/6, so if you created files or users since then, you will have to recreate them again. – Dan J

Update Mar 5th, 4:06 PST: The restore process continues, we appreciate your patience. If you are seeing any problems with your sites please contact support. Please note, the backups are from 1/6, so if you created files or users since then, you will have to recreate them again. – Andrea

Update Mar 5th, 9:50am PST:  The restore process continues. The load is still higher than we would prefer on the server, causing some slowness, that we are looking into. If you are seeing any problems with your sites please contact support. – Justin

Update Mar 4th, 8:45pm PST: The restore process continues to execute without any issues on this server. If you are seeing any problems with your sites please contact support to have it investigated. We can also check if your user was restored from backups, just let us know. – Andrea

Update Mar 4th, 4:00pm PST:  The restore process continues to execute without any issues on this server. If you are seeing any problems with your sites please contact support to have it investigated. –dan w

Update Mar 4th, 6:30am PST:  The restore process continues to execute without any problems. If you are seeing any problems with your sites please contact support. –JJ G

Update Mar 4th, 1:15am PST:  The restore process continues to execute and we’re not showing any problems with it. Our systems administration team is still working with the raid array on the old hardware and we’ll give more information on that when it becomes available. In the meantime if you are seeing any problems please  contact support. –JJ G

Update Mar 3rd, 7:51pm PST:  The restore process is still humming along, and we’re still investigating the old hardware’s raid array. We don’t have any new info on that front. We apologize for this. Again, if you’re seeing problems, please contact support –Brian H

Update Mar 3rd, 1:00pm PST:  The raid array is still rebuilding on the old machine. Once it’s completed we will either move services back to the old hardware or we will start to restore all up to date data. This depends on the health of the raid array when it’s finished building. As a reminder, we are aware that some of the data from the mounted backups may not be up to date. Once the raid array is rebuilt we can restore your current data.  While the process continues, if you have questions or problems please contact support. Thank you for your patience. -Oscar

Update Mar 3rd, 6:30am PST:  The  restore process on this host machine continues to execute and the machine has remained stable. While the process continues, if you have questions or problems please contact support. Thank you for your patience. -JJ G

Update Mar 2nd, 8:07pm PST:  The filesystem check on the old hardware continues to process. Some of you may have noticed that your sites are running off backups that are a bit outdated. Unfortunately we cannot restore data from that server at this time which is why we are waiting for the raid array rebuild to complete. Once the raid array is rebuilt, we will move all services back to the original machine and all of your current data will be restored. There is no way to speed this up, but as soon as it completes we will update this post. Please contact support with any questions! -Dan J

Update Mar 2nd, 2:30pm PST:  Some of you may have noticed that your sites are running off backups that are a bit outdated. The old server is having its raid array rebuilt, but the process is slow. We cannot restore data from that server at this time which is why we are waiting for the raid array rebuild to complete. Once the raid array is rebuilt, we will move all services back to the original machine and all of your current data will be restored.

The raid array is still building on the old hardware. There is no way to speed this up, but as soon as it completes we will update this post. Please contact support with any questions! -Oscar

Update Mar 2nd, 11:50am PST:  Some of you may have noticed that your sites are running off backups that are a bit outdated. The old server is having its raid array rebuilt, but the process is slow. We cannot restore data from that server at this time which is why we are waiting for the raid array rebuild to complete. Once the raid array is rebuilt, we will move all services back to the original machine and all of your current data will be restored.  We will continue to update this post as we progress with this restore. Please contact support with any questions! -Oscar

Update Mar 2nd, 6:50am PST: The new hardware is up and all apache instances are resolving. Backups have been mounted and the restore is running. At this time all sites are running from mounted backups. Please contact support with any questions! -Oscar

Update Mar 2nd, 5:55am PST: This server is currently offline for a file-system check, however we are taking the next step of moving it to new hardware as that fsck will take some time to complete. We will update this post once the new hardware is online and mounted from backups! Please contact support with any questions! -Patrick

This entry was posted on Friday, March 2nd, 2012 at 5:55 am and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

65 Responses to “Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups (RESOLVED)”

You can realize that there’s a problem in your hosting when you are visiting dreamhoststatus.com too often.
Was this planned?
How old is the backup?
It would have been nice to be notified in advance.

@Marc: No, this was not planned. The backup should be recent. Unfortunately there is no way to notify you in advance about a server going down as there really isn’t a way to predict when/if a server will go down.

@Oscar: Can you quantify recent? Hours? Days? Weeks?

I have since the start of January, after a few years of fine service, had extraordinary problems which Dreamhost have neither been able to or interested in solving.

Im on this port-au-prince and I sincerely hope it all get sorted now as moving my hundreds of websites would be a huge hassle, although thats what im planning to do long term as Dreamhost’s attittude has got the better of me.

So far in 2012 I have probably sent around 60 error messages and complaints, all of them via ticket support as the chat has been permanently down for two months. They never answer my questions, not even numbered and listed ones, and basically never tried to find out what the problem was but instead said it was probably something I did wrong in setting up my websites. Last week they also told me, after my frustrations grew that if im not happy with their services im free to bugger off.

My websites are down up to ten times a day, clients are outraged and im losing money. All Dreamhost have as response is that I should try to move my sites over to VPS which most liekly would solve the issue. As I said to them in one of the 4 mails I sent them this morning, I feel digitally raped by my server company, they are abusing me on a daily basis and laughing in my face.

It is intolerable.

Can you confirm if the backup is 2 months old?

im sorry but WHAT? are you guys really talking about whether this change will reset my hundreds of websites, some of them web portals, to days, weeks or even months back??????

Please dont tell me you are being serious?

My entire network of web portals are amon gother things based on foums and daily news updates, it would be the end of my business if this happened. We are talking about it being reset a few hours or to yesterday, right??????????

Also, in the message from Oscar 06.50, he says all sites are running from blabla… Well, I got two hundred sites and none of them are running, they havent been for hours.

This looks really professional, I will be a legend in my area after this:

Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”

Hi!
You have restored an old backup!!!!!!

Here my problems:
– I have lost my last week of work
– My account was hacked and you have restored the hacked files
– I have move some domain under different server’s user but your backup is older and now I get “bad_httpd_conf” error.

What can we do?

IS there a problem with MySQL Server ? I cant access any of my databases from my sites or Phpmyadmin…

it says in the messages above to contact support for support but ive done this without response all day

this is like a two month long nightmare which just got ten times worse just when you thought you had already hit the bottom

there is no way im EVER gonna stay a dreamhost customer after this, now I just want my sites back online to save my business and then I will start immediately to plan my escape from this sinking, burning and pestinfected ship

I pretty sure that Dreamhost has a probleme with databases since 3 hours…

I’m on “didgeridoo:youssef” MySQL Server

1. My websites can’t access their databases :

Critical Error
Error message: Cannot connect to the database.
From class: MySQL

2. I can’t connect to any phpmyadmin either :

phpMyAdmin –
#2013 – Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111

3. Can’t create new database from Dreamhost Panel, i’ve the followinf error :

“INTERNAL ERROR CREATING DB: connect_admin failed for didgeridoo:youssef. Please let support know!”

So, i think the problem isn’t my fault… Anything else I could try to be sure ?

This is really awful.
When I was beginning to think that I was the only one with a 2-month-old backup, I come here and realize that the pain is, just like port-au-prince itself, shared.
At this point, I have lost 2 months of work that spread all the way from contact forms to cgi scripts and time sensitive changes. If this isn’t what they call “disruption”, I’m afraid I don’t know how to name it.
Dreamhost agentes, will you please, please let us know what is *really* going on?

@Ale, @Tomas, @Gustavo: The latest update addresses your concerns about the older backups. We’re aware of the issue and waiting on the old machine to finish rebuilding it’s raid array.

@GobiDigital: This post is only for the shared server port-au-prince. The admin team however is working on your mysql server, but this post is only for issues related to port-au-prince.

I have gone through the entire day without any response in support, although Oscar keeps repeating to contact them via support.

Disgusting.

I still cannot form any opinion regarding how old the backups are or will be, actually I havent got any clue whatsoever about any of this apart from what I read from people on this very page and I find it absolutely and utterly appalling, a lack of respect for their own clients which cannot be excused.

I repeat, I have hundreds of paid business sites which all have been completely down, as if they simply dont exist, all day, for about 9 hours now. During these 9 hours, although I have contacted them numerous times, I have heard absolutely NOTHING from Dreamhost apart from what I read on this page.

It is just unbelievable, im responsible for web portals and websites for businesses, public offices, organizations and as all of them have been down for more than 9 hours I have heard nothing from my server provider and have had to lie to everyone to not look like a fool or even getting sued…

This is the 4th. serious outage of all my sites on port-au-prince in the recent weeks :-(

When is ps47533 going to be taken care of???!?!?!? It’s on a different homie, but your status posting for that homie says the issues with it have been resolved so I assume you’re ignoring my comments there. My VPS (ps47533) was down earlier this week, then came back up with really old files, and now it is completely hosed. It doesn’t have a /home, and all sorts of stuff isn’t working. I’ve filed OMG priority support requests several hours ago and NO RESPONSE. I’ve tweeted your CEO, with NO RESPONSE. What is the deal?

I’m on didgeridoo:knavel and having the same problem as GobiDigital. Can we get a status report on that mysql server?

By the way, my backup was from February 16th, (16 days old)

MORONI (webserver) IS DOWN TOO !!

@Flavio Moreira: the Apache service for your site was rebooted and it’s online now

Can you give us an outlook when the current data will online again? How far is the progress of rebuilding the RAID?

Thanks
Marcus.

all my sites have been down for 24 hours now.

Can anyone please say how much time they expect this to take? I got 3 people here at work sat playing with photoshop as we cant do anything.

I’m leaving port-au-prince, I can’t take this anymore. Good bye.

@Tomas You talk about change server or hosting company?
I ask because I’m evaluating both solutions…

I had a new site golive one week ago and now (due to old backups made when the site did not existed) they are offline… Really great; the custumer will be happy of his new hosting -_-

@matteoparrucci, I have spent all morning and last night investigating hosting companies to prepare a complete escape from DREAMHOST for the following reason:

– After one and a half year without other problems than occasional downtime, I suddenly after Xmas started experiencing insane problems with all my almost 200 websites – based on what I witnessed myself and what clients and users notified me about via Email and phone – being down from an estimated 3-4 and up to twenty times a day. Most of these downs would be only a minute or two although they could at times be up to half an hour.

As the chat since Xmas have been, at least to me, down without exception, I started sending an evergrowing number of error messages and requests to fix the problem. My main issue is that Dreamhost havent only been unable to sort this out but more than anything a weird and confidencebreaking combination of incapable and unwilling.

Most of the times, and we are talking about like 60 error mails and following replies, they have simply said that they cant see that the sites even have been down and signed off replies with “It seems to be ok now, if you experience similar issues feel free to contact us again”, while other times they have come up with things like that it probably is my location and had me tracing my routes. When I had people inn Norway, Japan, India and all over the world visiting and confirming my sites being globally down when down, they changed their theory to my websites probably being badly optimized.

I have felt very badly treated because I have over and over again requested a solution, with Dreamhost not only constantly trying to wipe themselves of responsibility that my clients websites are constantly down but also turning the blame on me while they clearly state that they actually dont know for sure why this happens. The only solution they repeatedly have come up with – without guaranteeing at all that it would solve anything, only saying “that will probably fix this” is to upgrade to VPS and pay lots more!!!!

Pardon me, you cant even have my websites online, and you want me to pay ten times more? I would be very happy to pay for a VPS, dedicated or whatever as long as I could be assured that it would work well, but when a company vioalte me as a client I cannot possibly be confident that this is the case. Also, for my web portals this is the solution I had planned all along anyway, but I have lots of small business sites with only a few visits a day and to register a vps for those would be foolish and waste of resources.

The problem is that even a hairdresser or car mechanic who only need a page as a reference point will be pissed off his clients repeatedly tell him they visited his page and it was offline. I havent really been unfair to Dreamhost – I have just asked for my websites to be online and for them to help me when they arent. Not hokus pokus really.

So, when i ask straight questions over and over again and they ignore them, I get fed up. For example, they say some of my sites have exceeded the “memory limit” – a concept which i didnt know even existed, and no matter how much I ask them what it means, what this limit is etc they never answer anything regarding this. I send a message asking “Please explain to me what memory limit means, how to see how much memory im using and what the actual limit you refer to is” and they respond talking about totally different stuff without even mentioning memory limits.

I have tried everything from explaining that I am considering a lawsuit to the fact that im considering dedicating part of my leisure time to create a website, facebook and/or twitter to warn people against their services but to no avail, they keep talking the same talk and do nothing. They completely ignore any educated request for professional assistance, angry and threatening shout or desperate drowned-in-tears plea for help in the same way – ignoring what I say and dragging things out while doing nothing.

Last week they even told me that they found it strange that I was only on shared hosting if these pages are so important to me, telling me that my many problems basically were due to being a cheapscate!!! And, incredibly enough, the company who neither have been able to address my problems or even respond to questions for months told me that if im not happy with their services im welcome to bugger off….

So yes, im 100% certainly a leaving customer.

Great, now even https://panel.dreamhost.com/ is down…… :-)

Could this be more patethic???

Just for amusing all of you as we are sat here staring at this page, through tearful eyes;

“Our expert technical support representatives are available 24/7 via email, live chat, and you’ve even got the option of reaching a human being over the phone. Our 100% uptime guarantee means we take uptime seriously.”

Bearing in mind that chat has been down for two months, support take hours to respond and then dont answer what you asked for and my websites have been down up to ten times a day so far this year, this statement is pretty senseless to me :-)

@Tomas It breaks my heart reading your and others desperate statements.

We all obviously complained several times to the support stuff about all kinds of down times on this server without really taken us seriously. Read responses to my messages below, it’s always the same. ” I’ve taken a look at
your sites and they are now loading quickly and without error!” (18 hours later!)

Since December 2011, the hosting service of dreamhost cannot be called reliable.

What I find really bad is that I did not get any notification about this out-times. So I cannot trust dreamhost and must install some kind of watchdog an another server to notifiy me about issues. This all makes me sick.

My support history:

On Fri, 03 Feb 2012, you wrote:

> our webserver ist down. Please check.
>
> [Fri Feb 03 12:37:10 2012] [emerg] mod_fcgid: server is restarted, pid 1002 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 17602 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 18937 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 17441 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 18101 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 17705 must exit
> [Fri Feb 03 12:37:13 2012] [emerg] mod_fcgid: server is restarted, pid 18587 must exit
> regards,
> Thomas

I’m very sorry that you’ve been experiencing issues. I’ve taken a look at
your sites and they are now loading quickly and without error! The log
that you noted regarding the server is restarted is just a notification,
it is not an error.

On Mon, 06 Feb 2012:

> it’s happening again.
> Error 324 (net::ERR_EMPTY_RESPONSE) on HTTP Traffic
> on all my domains over different networks.
> I’m starting to get nervous. Ping works very fast. SFTP is OK too.

I’m very sorry that you’ve been experiencing issues. I’ve taken a look at
your sites and they are now loading quickly and without error!

@Tomas: We’ve had a large amount of support coming in the last two months which has really caused us to get behind. We’re still trying to catch up which is why it’s taking us a little longer to get back to your tickets. Live chat is going to be turned back on starting next week and will stay on. I will be responding to your tickets personally.

@Thomas: That is the exact reason we have this status page in case you have an outage. We will also have a email notification system very soon.

So, all our websites went down and Dreamhost claim it was unexpected and left them no chance to notify us first… Make up your mind yourselves, below I have posted all REPORTED MAJOR OUTINGS for the last 10 weeks – notified downtime (17 in this period) and other downtime which isnt notified on dreamhoststatus.com is not included.

There are 108 in total, compared to 17 notified ones, making rougly 10 major outings a week! Now, what we need to ask ourselves is not only if DH’s claims that the outings of Castor and Port-au-prince the last two days can possibly be called “unexpected” or if Dreamhost should be allowed to falsely sign up new clients using a “100% uptime guarantee” statement on their website, but whether they are worth all this bother at all.

Downtime Dates & headlines of major outings (only) from dreamhoststauts.com

14.12.2011 – Web VPS host ‘homie-vserver81′ offline for hardware issues
14.12.2011 – Shared web server baham offline for RAID card replacement
14.12.2011 – Web VPS host ‘homie-vserver24′ being restored from backups on new hardware
14.12.2011 – Web VPS hosts homie-vserver265 and homie-vserver286 offline due to network issues
14.12.2011 – Network issues in Irvine datacenter causing delays to server updates
14.12.2011 – Shared web server ‘sainthelier’ offline for unscheduled file-system check
15.12.2011 – Shared web server ‘astana’ back online on old hardware
15.12.2011 – The minkar server is under DDoS attack
16.12.2011 – Shared web server ‘warriors’ offline intermittently due to Denial of Service Attack
17.12.2011 – Shared web server ‘johnson’ restoring from backups on new hardware
17.12.2011 – Shared web server ‘hagatna’ down for system file check
17.12.2011 – Shared web server ‘floyd’ being moved to new hardware
19.12.2011 – Shared web server ‘astana’ restoring from backups on new hardware
20.12.2011 – WebFTP temporarily offline
20.12.2011 – Web VPS host ‘homie-vserver147′ down due to hardware issues
20.12.2011 – Web server ‘porto-novo’ offline intermittently due to denial of service attack
21.12.2011 – Web server ‘porto-novo’ offline intermittently due to denial of service attack
25.12.2011 – Shared web server ‘tech21’ restored from backups on new hardware
25.12.2011 – Shared web server ‘minkar’ was under DDoS attack, one apache getting new IP
26.12.2011 – Networking issues affecting a row of machines at our LAX datacenter
28.12.2011 – Irvine datacenter networking issues
29.12.2011 – Intermittent downtime on shared machine ‘Wellington’ due to RAID issues
29.12.2011 – Panel down
29.12.2011 – ‘Manage Account’ area of Panel down
29.12.2011 – Shared web server ‘betria’ offline due to unscheduled RAID card
30.12.2011 – Shared web server ‘alcyone’ restoring from backups on new hardware
30.12.2011 – Web VPS host homie-vserver175 offline for unscheduled file-system check
30.12.2011 – DreamHost Panel offline
31.12.2011 – Web VPS host homie-vserver140 offline for unscheduled file-system check
02.01.2012 – Shared web server “crater” moving to new hardware
02.01.2012 – Messages originating from Webservers are being delayed
02.01.2012 – WebMail ( Squirrel Mail ) minor outage
03.01.2012 – Ongoing networking outage in one of our facilities
03.01.2012 – Email Relaying Delay
04.01.2012 – Webserver VPS “homie-vserver150? offline for unscheduled RAID card replacement
04.01.2012 – Webmail issues being investigated
04.01.2012 – Homiemail-sub5 mail service expansion
05.01.2012 – Web VPS host homie-vserver174 is being moved to new hardware
05.01.2012 – Shared web server ‘miaplacidus’ down
06.01.2012 – web server “tabbuts” is currently down.
07.01.2012 – Shared web server ‘mcgurn’ down for unscheduled file system check
07.01.2012 – Web VPS host homie-vserver190 being moved to new hardware
09.01.2012 – Shared machine ‘ Godin ‘ is moving to new hardware
09.01.2012 – Panel down
10.01.2012 – WebFTP outage
10.01.2012 – Shared web server ‘manama’ being moved to new hardware
11.01.2012 – Shared mail service issues on homiemail-sub5 is being investigated
11.01.2012 – Shared web server ‘hachiman’ offline due to unscheduled RAID card replacement
12.01.2012 – Shared web server “yangon” down due to hardware issues
13.01.2012 – Shared web server ‘dodoma’ offline for RAID card replacement
13.01.2012 – Network Outage affecting one of our Data Centers
13.01.2012 – Intermittent Stability Issues on WebMail
14.01.2012 – Shared web server ‘porto-novo’ under DDoS attack
15.01.2012 – Shared web server ‘tygra’ moved to new hardware and restoring from backups
17.01.2012 – Shared web server ‘dobro’ moved to new hardware and restoring from backups
17.01.2012 – Web VPS host ‘homie-vserver154′ moved to new hardware and restoring from backups
18.01.2012 – Shared web server ‘samick’ offline for RAID card replacement
18.01.2012 – Intermittent Webmail Stability Issues
18.01.2012 – Emergency Maintenance with our Easy one-click installs
18.01.2012 – Web VPS host ‘homie-vserver134′ moved to new hardware and restoring from backups
18.01.2012 – Major network outage
18.01.2012 – Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups
18.01.2012 – Emergency network maintenance
19.01.2012 – Hambone MySQL Downtime Due to RAID Issue
19.01.2012 – Intermittent Mail Instability on homiemail-sub5
20.01.2012 – Shared web server ‘portvila’ moved to new hardware
20.01.2012 – Changing Shell/FTP Passwords due to Security Issue
21.01.2012 – Web VPS host homie-vserver198 moved to new hardware and restoring from backups
21.01.2012 – Password changes delayed in the web panel
22.01.2012 – Web VPS host homie-vserver233 moved to new hardware and restoring from backups
22.01.2012 – Shared web server ‘albali’ under file system check and being moved to new hardware
22.01.2012 – Degraded performance on personal backup server hanjin
23.01.2012 – Shared web server ‘torr’ moved to new hardware and restoring from backups
26.01.2012 – Connectivity issues to a small subset of web servers
29.01.2012 – Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down
29.01.2012 – Webmail login issues
30.01.2012 – Easy/Simple One Click Installs temporarily offline
30.01.2012 – Connectivity issues in one of our datacenters
30.01.2012 – Shared web server Sinistra having issues
02.02.2012 – Email connectivity issue in one of our datacenters
02.02.2012 – Shared web server ‘portvila’ not accessible
03.02.2012 – Web VPS host homie-vserver168 being moved to new hardware and restoring from backups
03.02.2012 – Web VPS host homie-vserver163 being moved to new hardware and restoring from backups
04.02.2012 – Connectivity issues with our Irvine Datacenter
07.02.2012 – Mail bouncing for subset of customers
09.02.2012 – Web VPS host homie-vserver206 being moved to new hardware and restoring from backups
13.02.2012 – Web VPS host homie-vserver158 being moved to new hardware and restoring from backups
14.02.2012 – Mailservers on Trend Micro RBL, working on removal from list
17.02.2012 – Web VPS host homie-vserver197 restoring from backups to new hardware
17.02.2012 – Web VPS host homie-vserver173 offline for file-system check
17.02.2012 – Shared web server ‘paramaribo’ offline for RAID card replacement
18.02.2012 – Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware
21.02.2012 – Shared web server ‘cayenne’ restoring from backups to new hardware
22.02.2012 – Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware
22.02.2012 – Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups
23.02.2012 – Web VPS host ‘homie-vserver131′ back up and running on original hardware
23.02.2012 – Shared web server ‘stampeders’ restoring from backups
23.02.2012 – Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups
24.02.2012 – Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups
25.02.2012 – Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups
27.02.2012 – Web VPS host homie-vserver142 moving to new hardware and restoring from backups
27.02.2012 – Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups
28.02.2012 – Shared web server ‘mystique’ moved to new hardware and restoring from backups
29.02.2012 – VPS host machine ‘homie-vserver288′ under DDoS Attack
29.02.2012 – Connectivity issues in one of our datacenters
01.03.2012 – Shared web server ‘singapore’ moved to new hardware and restoring from backups
02.03.2012 – Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups
03.03.2012 – Shared web server ‘castor’ moved to new hardware and restoring from backups

Oh, and dont forget, Dreamhost claim the port-au-prince and castor outings were U N E X P E C T E D :-)

Ticket #4791647 opened 36 hours ago. This site has been down for 48 hours now without a single rssponse to my ticket… For privacy reasons I’ve changed the real username with “username”

Ticket text:
I’ve managed to restore most of my sites… BUT

This user has disappared from the Hard disk: “username”
The home (/home/”username”)does not exist anymore so the site is not accessible…
End of ticket text

Congratulations for great support and reliable hosting -_- the only pity is that I’ve payed the renew one week ago.

Hello,

Can anyone please give me an outlook how long the outage of my sites will continue? If it is about rebuilding/checking the raid array, those processes usually have some kind of progress bar. I won’t nail anyone down to the minute, but I need to understand what to expect: hours, days, weeks, months?

You need to give me something realistic to pass on to my clients!

bye
Marcus.

@marcus, @matteoparucci

when they finally responded to my support tickets after like 40 hours down, they have told me it could take another few days and that the backup will be from between 16 and 60 days ago.

They also confirmed that any work as well as any new website done within the last couple of weeks will be completely lost. On my own account, all work since 6 weeks ago seems to have been lost forever, among that ten new websites and one entire web portal.

Welcome to Dreamhost.

If it is any help to anyone, I was wise enough years ago to separate domains and hosting for all my sites from startup, meaning that I at least can set up accounts with any hosting company and have my sites somehow online again with at least an apology poster pretty fast. I urge anyone reading this to keep domains and hosting separate too, as it seems to be the one thing which saves me from possibly getting sued by multiple clients the coming week.

@INSANITY???: We have 1000’s of servers, while the list does appear large, not every outage affected all or even a fraction of our customers. I do apologize for the customers who are on port-au-prince who happened to be affected by this issue.

@matteoparrucci: I just replied to your tickets.

@Marcus: Are you on port-au-prince? The new server hardware is live and running from backups. The old hardware is currently being worked on, and it is still not known if it will be recoverable.

@Tomas: Unfortunately that is the case with recently changed data. We are working on improving the backups and how regularly they can be ran, with new hardware being deployed, changes in monitoring, but it isn’t an overnight fix.

Sorry, but at this time were still trying to recover data? It is important because I have a blog yet important down!

@Raffaello: I replied to your ticket a moment ago. Yes, that is the situation.

@Justine: Yes, I am on port-au-prince. The backup data is was outdated and non of the sites can be considered up until the current data is restored!

@justin Loading files back. Loaded an index.html but http://www.idea4usonly.com/ still gives error

@Justin now it seems to work… But still with 2 weeks old PERSONAL backups…

I have waited all day for response to this:

My old websites are now online but all the ones which were created since 6 weeks ago are gone. Now, is this it or is there more to do before I can start creating the missing websites all over again?

I of course ask because tomorrow is monday and im sat here like an idiot doing nothing, not knowing whether I can start designing these pages with the bits I used before now, or if I should wait as whatever work i do now could be void again later tonight or tomorrow?

Can anyone tell me if this is it? Am i good to go, can I spend the night trying to get stuff up and running or will it be a waste of time?

I think as Tomas. Grazie

Wait until the end of operations could be a good idea?

We got and important work project on that server (now lost) any chance the data could be restored from a more recente backup? Like 29 Feb 2012 ?

Ma sul quel server c’erano solo italiani??? Era un server di sfigati???

Io domani perderò molto probabilmente un cliente molto importante. Sono arrabbiatissimo!!!
(Tomorrow will most likely lose a very important customer. I’m angry!)

I remember the times when prices, speed and live support where important factors when choosing hosting company

these days just having your website online and intact seem like a privilege…

:-(

@Filippo: Unfortunately at this time we don’t have more recent backups. I put your ticket with the others from port-au-prince so they can receive an update when or if we can repair the old hardware. I wish there was anything more I could do to get you the data, but it is in the hands of RAID experts and the hardware status.

@Raffaello: I believe I have responded to you today, if so, you will also receive a response on the situation of the old hardware. The backups are what you see now, whether you are running from them or they are restored to the server.

I do have a serious question though, which I would like Dreamhost to answere here, in front of everyone;

What are your views regarding economical compensation for those who have suffered big and small economical losses because of this?

Do you take responsibility and compensate people, or is having entire websites wiped out, losing six weeks work, losing clients and even getting sued considered by you as something any customer simply has to accept as part of the game and an unavoidable risk when being webdesigner?

please need to know if my data will be recovered, it is imporatante also know when it will be ok

@Tomas – this is and has already been posted in our terms of service: http://dreamhost.com/terms-of-service/ – Material Products #11. The domain you linked to your comment is not hosted with us. If you were affected by this outage on another domain, please send a ticket through the DreamHost panel, so we can discuss your specific situation.
@Julio – we’re restoring your data from backups. Two of your users are yet to be processed, but the rest are done. I’m replying to your email.

Can I be informed of the last backup available / when will it be restored ASAP? Thanks

I recommend people here to follow the debate and get interesting info on the thread for the castor server failure, this way I wont have to double post here.

@andrea, you say if I was affected please contact support :-) I assume that you just came to work after a relaxing weekend nad havent read any of the stuff on these pages, so I will excuse you

“…send a ticket through support…” hehehe, thats funny :-)

Hello @andrea
Can I know if my users have been processed or when will they be?
Thank you

@Marc: The user for the domain in your comment, is on local disk now, so has been processed.

@Tomas: Please do not double post or cross-post. We have been very allowing of your large number of comments I feel, even approving the comments that the spam filter in wordpress put in moderation. We don’t want it to be necessary to start doing further restrictions due to abuse.

@Julio Contreras Maluenda: Your restore from backups is complete. What you currently have is all that we have available in backups. Databases were not affected by this outage, if important data was stored in there.

I have been working a few hours on setting up the first of the 11 websites I lost in this unexpected meltdown, suddenly it is gone again….

@justin, I have lost more than ten thousand dollars in two days purely because of your lack of professionalism, if you feel you have been very allowing I think you should put yourself in my place…? Also, im not spamming anything, im asking for some straight answers on behalf of everyone, like to why you havent got more people to deal with a problem and support pressure which have been like this for almost a year. The reason why im repeating these questions is not spamming but the fact that you dont answer them properly but instead talk yourself away from the subject.

I mean, lets do a test here? Question: Every day for a year more or less, you have had this problem, with servers going down unexpectedly and people going mad and bombarding your support inbox while you take hours and days to respond. Why havent you got more people then to deal with it, when you know very well that it will be like this?

You may argue that it is, individually spoken, unexpected when a specific server goes down with the same problem, you may call it unexpected that THAT server went down, but what you cant say is unexpected is that SOME server goes down when it happens every day for a year, so then you should have staff to deal with it.

I mean, Justin, just answer this question and woopie, there is one less thing for me to “spam” yo uabout…right? Yes?

@Tomas: A single server having an issue at a time we have had in the past, only the last few months have been exceptional in more of these issues. We have all our staff working on the issues, and we are always hiring more people.

I did not say you were spamming, I said the wordpress built-in spam filter and plugins thought you were spam. I only asked you to not cross-post to other threads not related to your issues. For example in the homie-vserver172 post, that server has recent backups and no data will be lost. Telling customers in that thread that they will lose data or have extremely old info is incorrect and not helping them.

Hi, there are problems with FTP connection. Confirm that you are not deleting anything?

@Raffaello: those FTP connection issues were likely due to our datacenter connectivity issues earlier, I’m now able to connect to your FTP user

One of mi sites, y pay a company for upload 350 products on my catalog. So thank you dreamhost i need to pay again.

What a Shamefaced.

Andrés

Please add my address to the ticket as well. Will recent data be overwritten if a newer backup becomes available? I can’t go through this process twice..

Also, since the restoration, this server is very unresponsive. Load times of 60 seconds or more on all domains hosted on this server.

I moved from port-au-prince to a VPS on Sunday, which turned out to be a good decision if I watch at this deseaster. The site is now very responsive. I managed to repair user rights on directories due to the sysadmin account, but there are still issues. I have no FTP connectivity. The user account with all the sites is still somehow linked to port-au-prince, and after adding a new user to test FTP I lost the login to the first one. Thank god I still have the sysadmin login. Panel down hours ago. Support? No real help in 12 hours. What happened to dreamhost?

Due to the ticket system is flooded. My TIME CRITICAL TICKET about a domian CANCELLATION ISSUE is being ignored. Please help !!! This is REALLY TIME CRITICAL! Thank you.

The TIME CRITICAL ticket:
#4799432
#4799173

@Michael D: There will unfortunately not be a newer backup available, so please go ahead and restore the data and it will be safe from being overwritten.

@Thomas K: There is not enough information in your post to look up your account, so I cannot comment.

@Neo: Your issue is an abuse issue. Only our abuse team can handle those. Sorry I cannot be of further assistance and will have to ask for your patience.

Just to tell what happened after previous post, I found out that the user not existed on the VPS. I decided to move all domains of this via ssh and sudo mv command to a new user, to set the user rights to the new user via “sudo chown -R : and final set the domains in the panel to the new user.
Everything is up and running. Still think it was a good decision to move to a VPS.

hello
your restoration has ruined my site and it is being published now with wrong pages.
please restore it to mid January dates ASAP.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘singapore’ moved to new hardware and restoring from backups (RESOLVED)
9:16 pm

Shared web server ‘singapore’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 1st, 2012 at 9:16 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘singapore’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your Web Panel.

Please note, this is regarding the shared web server ‘singapore’, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 6th, 4:20pm PST: The restore on this server is now complete. We are now resolving this status post. Please contact support if you have any further problems on this machine or have any other questions or issues! -dan w

Update Mar 6th, 1:30pm PST: This restore is moving along still and all services are reporting online at this time. Please contact support if you have any questions, or issues! -Brian S

Update Mar 6th, 10:15am PST: This restore is moving along still and all services are reporting online at this time. Please contact support if you have any questions, or issues! -Justin

Update Mar 6th, 4:58am PST: This restore is moving along still and all services are reporting online at this time. Please contact support if you have any questions, or issues! -Patrick

Update Mar 5th, 9:00pm PST: We’re continuing to restore your sites from backups. Everything should be up and running. If you need assistance, please contact support to let us know. – Dan J

Update Mar 5th, 4:12pm PST: We’re continuing to restore your sites from backups. Everything should be up and running. If you need assistance, please contact support to let us know. – Andrea

Update Mar 5th, 10:00am PST: We’re continuing to restore your sites from backups. Everything should be up and running. If you need assistance, please contact support to let us know. – Justin

Update Mar 4th, 9:00pm PST: We’re continuing to restore your sites from backups. Everything (based on those backups) should be up and running. If you need assistance, please contact support to let us know. – Andrea

Update Mar 4th, 4:00pm PST: The restore is running without issue and still haven’t seen any problems with the Apache services running the site. If you are seeing any problems with any of your sites, please contact support to let us know. –dan w

Update Mar 4th, 7:30am PST: The restore process continues to execute and we are showing no problems with the Apache services running the site. If you are seeing any problems, please contact support. –JJ G

Update Mar 4th, 1:45am PST:  No new issues to report, the restore process continues to execute. However, if you are seeing any problems, please contact support. –JJ G

Update Mar 3rd, 7:51pm PST:  The restore process is still humming along, so no new issues to report. Again, if you’re seeing problems, please contact support –Brian H

Update Mar 3rd, 1:35pm PST: The restore is just about done. We expect it to finish very soon and we will update this post as soon as it does. Please contact support if you are experiencing any trouble! -Oscar

Update Mar 3rd, 11:00am PST: This restore process continues to execute without issue. The server remains stable and we are showing no problems at this time. Please contact support if you are experiencing any trouble! -JJ G

Update Mar 2nd, 7:59pm PST: This restore is still moving right along! The server is running well and we are receiving no reports of issues with the hosting. Please contact support if you are experiencing any trouble! -Dan J

Update Mar 2nd, 1:15pm PST: This restore is running without any issues.  The restore is more than half way done. We are receiving no reports of issues with the new server and the mounted backups. If you have any issues at all with your sites,  please contact support with any questions! -Oscar

Update Mar 2nd, 4:49am PST: This restore is still moving right along! The server is running well and we are receiving no reports of issues with the hosting. Please contact support if you are experiencing any trouble! -Patrick

Update Mar 2nd, 12:26am PST: This restore is now running! The server has been online for about an hour and all Apache services are reporting online. If you are experiencing any issues at all, please contact support for assistance! -Patrick

Update Mar 1st, 10:44pm PST: Please note the changes to this post, as the server did not pass the file-system check and is now being moved to new hardware and restoring from backups. The server is still offline at this time as we prepare the new hardware. We will update this post with more information shortly! Thank you for your patience and please contact support with any questions you have! -Patrick

This entry was posted on Thursday, March 1st, 2012 at 9:16 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

10 Responses to “Shared web server ‘singapore’ moved to new hardware and restoring from backups (RESOLVED)”

Still down along with all our other sites – Please advise ASAP

@Adrian: You are hosted on a VPS, not the shared web server ‘singapore’. Your issue is completely unrelated to this post and is not associated with any other posted issues. Please Contact Support from your Web Panel for assistance with your downtime.

still not working for us!

@Noah: Your site is up and working for me. http://saleschallenger.exbdblogs.com/

For some reason, the way you guys do backups doesnt let me use most of vbulletin. The error when posting is

Fatal error: Class ‘vB_Cache_Db’ not found in public/vb/cache.php on line 128

so the forums are basically useless. Tried turning of cache, didnt work. Im just hoping the backup finishes.

@DWN: If you haven’t already contacted submitted a ticket through the Dreamhost panel I would suggest you do that. I’m not sure which site you’re referring to but I’ll try to get that issue figured out for you.

Still waiting for it to get fixed. The error is here: http://topnotchclan.com/content.php

@DWN – I checked your http://topnotchclan.com/content.php domain and it’s loading. Please contact support through the DreamHost panel if you need further assistance.

Yeah got it sorted, but now weird things are happening. Im missing random recent files and my vb directory was showing in FTP but was getting a does not exist error. I had to rename the folder and reupload the correct folder with the files. The backup is screwing with my current install.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Roundcube webmail upgrade tonight, March 1st, at 7pm (PST) (RESOLVED)
3:37 pm

Roundcube webmail upgrade tonight, March 1st, at 7pm (PST) (RESOLVED)

Posted (March 1st, 2012 at 3:37 pm PST) by danwy

We will be taking our Roundcube webmail interface offline tonight for a short period of time to upgrade its software. This will only affect your ability to login via the Roundcube webmail client. If you’re using a mail client like Outlook or Thunderbird, or the Squirrelmail webmail interface, you will not be affected. We will update this status post when the upgrade is complete or if there are any unforseen issues that come up. Please keep an eye on this post for updates and contact support with any concerns or problems.

Update Mar 1st, 7:56pm PST: The upgrade is now complete. The RoundCube installation has been upgraded from from version .5.3 to .7.1. This should boost performance as well as allow for some fancy new features. For a full list of changes you can check here. Thanks -Dan J

This entry was posted on Thursday, March 1st, 2012 at 3:37 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

One Response to “Roundcube webmail upgrade tonight, March 1st, at 7pm (PST) (RESOLVED)”

Comments: 0 ?

Not anymore!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues in one of our datacenters
2:05 pm

Connectivity issues in one of our datacenters

Posted (February 29th, 2012 at 2:05 pm PST) by andrea

Some of our customers may be experiencing connectivity issues right now. This would affect web and mysql services, but only for a small subset of our customers. We apologize for the inconvenience this is causing, and we will be posting updates as soon as we have more information. – Andrea

Update Feb 29th, 2:18pm PST: We are happy to confirm that the situation was quickly identified and has been resolved. There was a slight misconfiguration with our external routing, but it has been corrected. We’re also continuing to investigate to make sure this will not happen again. Sorry about any inconvenience this may have caused. If you’re still experiencing connectivity issues, they are likely unrelated, so please contact support with the details. We’ll be happy to assist. – Andrea

This entry was posted on Wednesday, February 29th, 2012 at 2:05 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

55 Responses to “Connectivity issues in one of our datacenters”

Is there another Irvine problem? Dedicated server ds2286/spica here..

Please help us!

2nd! My VPS is down…customers not happy. Help!

We’ve had issues all of february, so many that you moved our sites to another server. And now this! You keep adding $5 here and there to our account so that we stop complaining, but the truth is that we cannot run our business using such a unreliable ISP!

My porn site is down!!!! I’m losing money!!! People want to see naked girls doing yard work!!! PLEASE HELP!

Yep, we are down too! Not again after 4 days of outage. Fix asap

sasin (web server) and ukulele (mysql) are unreachable.

“…but only for a small subset of our customers.” Seriously? I seem to always be in that small subset!

spica here…

I just submitted an ticket, maybe this is it?

My site seems to be up for now but DNS is not working on my VPS…

Thanks Dreamhost! I’m back up!!

My VPS is also down, and has had intermittent service all month (the first month of hosting). This is not an auspicious start. I know you’re working hard, though.

Everything on ‘proxima-centauri’ web server and ‘lute’ mysql server is down.

I seem to be unable to connect to my SVN on nihal :(

My VPS is down again as well. This is killing us.

Back online. Thanks, DH.

Also lol @ ‘connectivity issues’ being a nebulous euphemism for ‘nothing is working’.

Pls my sites are down, more or less how long it takes to fix it?

Back online – good job DH!

Oh back here too. Safe.

sualocin for web
harp for mysql

i guess small subset isn’t so small after all…

Hope the problem gets resolved real soon. I have a client presentation of a website on the sadalbari server coming up.

@ All: We believe we found the issue and have resolved it already. We’ll be updating this post shortly with more info once we’ve confirmed things are fixed.

Back online here. Thanks DH!

The site on sadalbari is up and running. Thanks guys!

My site is backup as well. Thanks for fixing it.

February 29th, 2012 at 2:16 pmyou are a nightmare host Says:

Seriously guys. You are killing me. My site has barely been up for a day after a 4 day outage and now again its down. This is beyond unbelievable and I think I’m going to jump out of a window if is does not come back up soon. I am losing users, money and hope.

Site still down: ps81521

Back up! Thanks DreamHost

Dreamhost, I am disappoint.

You are causing more work for me each time there is an outage. I have started researching other hosting companies.

Is this the same db issue that I reported at 22:30PST on 28th? – still no feedback on my ticket.

Still down here

both my websites are down.

down down down!!

Back up! Thank you, that was a quick one- but please…no more

My sites seem to be back up. Hopefully yours are soon as well.

back now

back up and running, thanks was down a total of 54 minutes

Spica back up. Thanks for the quick response!

@ All: Per our latest update, this issue has been resolved already. If your sites are still down it is not related to this incident. If you’re still having problems with a VPS server, please check the rest of the status posts on this site to find the one that is directly related to your situation. This was a quick outage that lasted less than 20 minutes, so if there was history of the site being down the last few days (or if it is still down now) it was not related to this specific issue.

50 minutes of downtime due to a misconfiguration? How does that happen?

what happen? my page downtime :S help…yesterday and today i had problems

unable connect to my wordpress

Webserver: bruno
MYSQL: masevo

waratahs is unreachable from Europe and Japan. Ganbatte kudasai !

I have new dedicated server ds3152 and after 24 hours all my sites are still down! After the events of the past month my clients are ready to jump ship. Why is nothing working?

All of my websites went down for over two hours yesterday, we lost 4 clients totaling 6,000 / month. Woke up this morning with a dreamhost bill in my inbox.

we, too, are in the “small subset” – we are down again. ps52432

dreamhost panel is DOWN, too. How can we log a problem ticket!

PANEL IS OK, BUT WEBSITE IS THE OPPOSITE OF UP.

copyartwork.com web server and database is DOWN on ps54649

my site is so slow….what happen? anyone knows?

Website down, dreamhost panel down. First day of my free trial. Sad.

Seems like this is an issue again…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » VPS host machine ‘homie-vserver288′ under DDoS Attack (RESOLVED)
1:03 pm

VPS host machine ‘homie-vserver288′ under DDoS Attack (RESOLVED)

Posted (February 29th, 2012 at 1:03 pm PST) by danwy

Customers on the VPS web server ‘homie-vserver288’ may not be able access their server or sites due to an ongoing denial of service attack. Our admin team is looking into it and working on containing and stopping the attack. Our sincere apologies for any inconvenience this is causing.

Please note, this is regarding the web VPS host server ‘homie-vserver288′ and no other services (email, MySQL, shared) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

Update Feb 29th, 7:12pm PST: We found the specific VPS that was being attacked. It was shut down and the IP address was null (trash) routed, and the vserver host is now healthy once more. We appreciate your patience while we got this worked out! -Seohee

This entry was posted on Wednesday, February 29th, 2012 at 1:03 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

111 Responses to “VPS host machine ‘homie-vserver288′ under DDoS Attack (RESOLVED)”

Here is a list of affected servers:

ps58122 ps58132 ps58475 ps59150 ps59235 ps59404 ps59630 ps59688 ps59893 ps59911 ps60042 ps60252 ps60284 ps60359 ps60552 ps61013 ps61133 ps61282 ps61297 ps61312 ps62114 ps63277 ps63303 ps63367 ps63454 ps63477 ps63584 ps63618 ps65555 ps65603 ps65618 ps65634 ps65686 ps67675 ps67925 ps68079 ps68286 ps68293 ps70401 ps70657 ps73191 ps73230 ps73380 ps73697 ps73723 ps73735 ps82685 ps82824 ps82853 ps82976 ps83100 ps83147 ps83628

Could this be preventing messages from appearing in my webmail inbox?

Again my site is not working… This is a monthly routine now…

Followgram.me… down….

We stopped getting mail around noon. I went to web mail and got an error when trying to reload the inbox. It had an email address for dreamhost. Sent the error message to them but have heard nothing back yet.

you guys are killing me.

My site is down as well.

AGAIN my ecommerce site is down, thanks to dreamhost. More like nightmarehost. I’ve got orders to process and dreamhost’s lack of reliability is preventing me from transacting sales.

I’m on ruchba – why is ruchba being affected?

Dedicated servers are affected as well? None of my sites are accessible right now.

I am having the same problem receiving e-mail. I have not received an e-mail since about 11 am MST.

Thanks,
Jake

another dedicated server affected here

My website does not have a VPS and it is down. What’s going on???

I have started looking at different hosting, after 6 years.

I have two different VPS servers, and neither one is responding. FWIW, they are *not* on the list of psXXX servers listed above, so it seems that the issue is more widespread than was first indicated.

Dedicated server here too, and we’re completely down.

Appears to be affecting ‘sarin’ as well.

February 29th, 2012 at 1:37 pmNetrapal Singh Says:

Guys : you are doing funny with our work. Emails not working properly.

Why emails goes down again and again . Can you not provide us 99% up time for it.

You are killing me and our work. I write down email to you but no reply yet.

My VPS number is not listed above, yet my site is also down.

…now what. How long will it take to fix it?

Dedicated server down over here too

Looks like mothman’s been hit, too

our website is down too and we are not listed on the above list.

I control three accounts (with many domains) with Dreamhost.
2 no email (homie-master)

Panel is not accessible either. I beleive the DDoS is not limited to one machine.

I don’t have a VPS account, but my website is down. Any ideas as to why there’s a ddos attack?

‘nair-al-saif’ affected as well. I am down

the interesting thing is that some times ago dreamhost has communicated that they were working on security update against DDOS.. a very good result!!

Okay…dreamhost I am going to call Chairman Meow on you. Meh.

same email problems as Gary @ 1:39 pm on homie-master

We are on muphrid and our sites are all down

My recently launched API is unbearably slow as well. ps64596

I run my friends site treesofjoy.com which isn’t VPS (regular shared hosting) and its down as well.
Probably related to VPS issue above?

My site is down and I have no FTP access. It went down about 4:20 pm EST. Can you confirm that this is due to this attack?

Affecting “sarir” too, it seems. (Someone above said “sarin” but isn’t that a poison?)

Hi Guys, our VPS boxen PS60024 is not reachable, submitted a ticket as well. Getting a 502 bad gateway and 504

Well, it’s hard to tell if a new security update works until it’s tested. Hopefully they now have enough data to prevent future attacks ;-)

We are down too. We are a business and this is costing us.

I do not have VPS and my site is down. Looks like I am not on these servers either.

I’m on ps58717 and I’m down

ps61077 is down too.

My VPS is not listed, but is down! F@#$!

Should have said, I am on sarin too.

Any recomendation for another Hosting company more reliable?, its been worse all the time, pingdown is sending me emails all the time, up-down –up-down..
Now is down.

My vps server is also down; eight web sites down just like that. :(

Windows Azure is down as well – so I guess Dreamhost should feel they are in good company!

This ticket needs to be updated – the affected users certainly appears to be more than the original list of affected vps’s!

This is getting a little ridiculous. I’m happy to support a company that provides free web hosting for non-profits, but with a major site launch this Monday, my livelihood is riding on your servers. I’m seriously considering moving to 1&1.

wow, I just launched a site for a client and now its down. I regret recommending dreamhost to them. Man I look like a fool.

Not only VPS and dedicated too. Worse, the chat as off Dreamhost has problems! Absurd!

Dan H, yes, same for us – problem started around noon EST today. Can’t get any new emails and getting an error message in webmail.

Well, I reported our homie-master issue and the panel now shows this:

3 hours 17 mins ago: Outage verified: We are actively looking into resolving it.
3 hours 55 mins ago: Outage first reported.
Please note: These tracked outages have been reported by customers like you and the status is not automatically updated by our system.

February 29th, 2012 at 1:48 pmJason Evangelho Says:

We have a dedicated server NOT on this list, and we went down moments before Dreamhost posted this update. In the midst of 2 contests, paid advertising, and a ton of Vita launch coverage. Thanks Dreamhost!

Folks,

I am 100% sure that DreamHost knows about these server issues. Reading comments like, “I’m thinking of switching to a different host.” is ridiculous! Every web hosting company has issues at times. I am certain they understand that so many of us depend on reliable hosting for our online businesses, etc. but it doesn’t help to just complain. Let’s give them a chance to fix this problem and see what happens from there. Shall we? ;)

-DJ

Dedicated server spica is down.

ALL of our websites and client websites are down as well and the server is completely unreachable. Reboots from the panel have been absolutely useless… and the live chat support seems to simply no longer be staffed. Ever.

Also, our server is NOT on the above list. Considering the troubles experienced this past weekend, I’m thinking it may be time to find a new host. This is absolutely, utterly unacceptable and we’re the ones who are going to get burned at the stake by our clients for it. I expect the phone calls to start any minute. I am NOT looking forward to that.

Thanks, Dreamhost.

We have a dedicated server NOT on this list, and we went down moments before Dreamhost posted this update. In the midst of 2 contests, paid advertising, and a ton of Vita launch coverage. Thanks Dreamhost!

What´s going on?? Did DreamHost moved to Greece?

my server is not listed in the above but it’s down… any chance of getting an updated list of the affected servers??

:( My vps is in the list (a lucky day). I’ve reported an outage 20 min ago and support tool/team/dontknowwho said everything was fine :( I know it’s not Dreamhost fault, good luck guys! BTW my client will kill me in a few minutes :D

and perhaps a status update…

I’m not even signed up for a VPS, but all of my sites are down (including ones that I am currently paying ads for). I wish I could defend DH on this one, but I can’t find it in me. I’ve forgiven a lot of DH downtime and problems recently, but as customers we shouldn’t have to wake up every day and play a guessing game as to if our sites might go offline again. I still love you DH, but this is getting tiresome.

@David,

Another one of our VPS servers was down almost all weekend. This can’t happen. It literally devestates our business and our clients businesses each time it happens.

Shared ‘Suns’ server down as well. 2 sites offline. Far from just a single VPS issue here.

February 29th, 2012 at 1:57 pmCharlie Isaacs Says:

My site is down too

Shared Suns server down too. 2 websites offline. Definitely beyond a single VPS server issue here.

@DJ
You are of course correct that every hosting company has problems AT TIMES. Dreamhosts quality issues have recently been way above average. And it’s not like its been minor inconvenience downtime of a few minutes or so, just look through the history of dreamhoststatus.com!

It’s not like DH is reading every comment instead of working on fixing the issues.

My server is the ps33882 and once again is down. Can we know what is going on? And prevent it. Thanks

All sites down for us.

I’m not on the list, but i am having issues too.

Pinging ps63690.dreamhost.com [64.90.47.66] with 32 bytes of data:
Request timed out.
Request timed out.
Reply from 216.17.51.20: Destination host unreachable.
Request timed out.

Ping statistics for 64.90.47.66:
Packets: Sent = 4, Received = 1, Lost = 3 (75% loss),

Whoever is firing layzors is creating civilian casualties!!!!!
:(

All of my websites hosted on rana down. Hope to get a status update soon.

Where is the redundancy?

ps59368 is also down…again. sigh…

Web server Sadira down too. Nothingness…

Our server is not listed, but it is down as well. bambam ps59789

February 29th, 2012 at 2:04 pmJason Evangelho Says:

ds2231 is down. Going on an hour now.

ps80714 is also not listed, but completely down!

ps59701.dreamhost.com is down right now as well…

Somebody piss off Anonymous or something? Got nothing here on ‘Navi.’

It seems “regulus” is also down…

Suns must be down too as my site is offline too. Getting “500 Internal Server Error” now. It wakes up every now and again and then falls over.

To those complaining about reliability, and service issues, and “my VPS/DS/shared hosting is down too!” do remember that a DDoS attacks network resources, which may be shared across multiple systems and clusters. Odds are homie-vserver288 is the /target/ of the DDoS, and other systems are casualties of maxed out network pipes. So, blame the entities performing the DDoS, not the entity on the receiving end.

My sites are on nekkar and they are down too. Apparently they also went down last night for a while

Have 3 sites on 2 servers, all three are down.

Servers: spica & peacock both appear to be affect, neither one is VPS, both are dedicated.

sites are up here

after 50 minutes down

February 29th, 2012 at 2:11 pmJason Evangelho Says:

Ours just came back online.

It is up and running now… Thanks Dream host… issue should be fixed.

I’m back up now.

I give up. I will never find a reliable host. This is sad.

@Josh – I was thinking the same thing.

@Sehro – That’s all well and good, but considering we got an email on 2/11 with a letter from the CEO, Simon Anderson, stating the following after the LAST attack:

“In the process we identified a whole slate of improvements to our security systems, software and data management that we’ve already been implementing. Security is one of those issues that we can’t give too much detail on, but rest assured we’re on it and have upped our game substantially so that this kind of event is prevented in future.”

And: “I’m fully committed to ensuring we deliver a reliable, flexible and useful service for you going forward.”

You write, “So, blame the entities performing the DDoS, not the entity on the receiving end.”

No, I blame the entity that can’t keep it’s network secure. That’s not what I’m paying for.

ps59124 seems to be affected as well. Have not been able to access the server for the last hour. Email has been up and down, but seems to be functional now, but website access is completely offline.

Dedicated server is backup now.

I’m on ps59947 which I can’t even tell if that’s the homie server and my sites are totally down.

And…We’re Back!!

Hopefully you are ALL back online too.

Now, I just realized all of my WordPress sites have been compromised and most of the PHP pages have that Base64_Encode String of redirection on the first line AND I see a couple of files that I don’t remember being there at all before :(

@ ALL: This status post ONLY affects the VPS servers listed in the first comment above. If you are not on one of those VPS servers listed above, then you are not affected by this specific issue. If you have a VPS server, try checking the other posts on this status page to see if your server is listed on one of those. If you’re on a shared hosting server, we had another issue that we recently posted about which was likely the cause of your problem.

OK. my site is back live.

My site are on this server but they are NOT back up yet. I moved to a VPS for BETTER service but I have had nothing but trouble since moving. Ugh.

My Site is live now, too. Not bad for recovering from a DDOS attack.

lost all the email in my inbox and can’t seem to access web mail at dreamhost at all. Very frustrating.

Our vps is not listed but it’s been horribly slow all day. Oh wait, that’s normal.

Can’t see email in my inbox from the morning and afternoon; can’t access it via webmail either. Some email currently is coming in as replies to messages just sent

I can view the site in panel, but it can not work. When will back to work?

ps54088 down again.

My site is slow, is very slow =/

This is an Anonymous attack in retaliation for Dreamhost’s support of SOPA.

Shutting down the VPS is not a solution. You should fix the problem, not ignore it. They can easily target another VPS with the same attack.
And what about the paying customer that you shut down his VPS? How will you bring back his VPS and website to live?

I agree with Alon. Don’t negotiate with terrorists!

Seduce them.

oh this is unbelievable Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Come one Dh this is ridiculous. My sites were up for a day after 8 days being down. You delete my files, my copyrighted material and now they’re down again.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Refund me a years hosting. I could bill you for my time but that would be a lot more. And my loss more still.

Come one Dh this is ridiculous. My sites were up for a day after 8 days being down. You delete my files, my copyrighted material and now they’re down again.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Refund me a years hosting. I could bill you for my time but that would be a lot more. And my loss more still.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘mystique’ moved to new hardware and restoring from backups (RESOLVED)
3:12 pm

Shared web server ‘mystique’ moved to new hardware and restoring from backups (RESOLVED)

Posted (February 28th, 2012 at 3:12 pm PST) by brians

Due to issues with the RAID controller on the shared web server ‘mystique’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘mystique’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We will continue to update this post as information is made available. Please check back for updates.

Additional information: The mystique webserver was upgraded from Debian lenny to Debian squeeze during this hardware replacement process. Please review this status post for additional information if your rails sites are having issues with mysql2 gem or similar. – Justin

Update Mar 2nd, 1:40pm PST: Great news, the restore is done! If you are missing any data at this time, please contact support and we can check the backups for you  and restore any missing data.  -Oscar

Update Mar 2nd, 4:54am PST: This restore is pretty close to being complete, but still restoring at this time. All services are reporting online at this time, so please contact support if you notice any issues with your hosting! -Patrick

Update Mar 2nd, 12:41am PST: This restore is nearing completion! Just a few accounts left before we can mark this resolved. All services are reporting online, but please contact support if you notice any issues with your hosting! -Patrick

Update Mar 1st, 5:40pm PST: The restore process continues on this machine. All services are running and sites should be up and running. Pleasecontact support if you notice any missing or outdated data, or if you are experiencing any other issues. We’ll continue to update until the process completes! – dan w

Update Mar 1st, 8:40am PST: The restore process continues to run smoothly. All services are running and sites should be working. Please contact support if you notice any missing or outdated data, or if you are experiencing any other issues. We’ll continue to update until the process completes! – Justin

Update Mar 1st, 3:20am PST: The restore process continues to run smoothly, and we are still around 90% restored. All Apaches are up and running. Please contact support if you notice any missing or outdated data, or if you are experiencing any other issues. We’ll continue to update until the process completes! – Chalise

Update Feb 29th, 6:20pm PST:The restore is still continuing and going smoothly. The data is around 90% restored at this time, and Apaches are up and running. If you notice missing or outdated data, or experience any other issues, please be sure to contact support. More updates to come! – Seohee

Update Feb 29th, 3:08pm PST:The restore is continuing and going smoothly. The data is around 90% restored at this time, and Apaches are up and running. If you notice missing or outdated data, or experience any other issues, please be sure to contact support. More updates to come! – Seohee

Update Feb 29th, 9:15am PST:The restore is continuing and going smoothly. The data is around 25% restored at this time, and apaches are up and running. If you notice missing or outdated data, or experience any other issues, please be sure to contact support. More updates to come! – Justin

Update Feb 29th, 4:05am PST: The server is mounted from backups and the restore process is continuing to run smoothly. All sites should be up and running as well. If you notice missing or outdated data, or experience any other issues, please be sure to contact support. More updates to come! – Chalise

Update Feb 28th, 5:30 pm PST: Backups have been mounted to the new machine and all apache instances are online now. Your sites are now running from mounted backups and the restore has started. If you notice missing or outdated data, or experience any other issues, please be sure to contact support.  We will continue to update this post as the restore progresses.  – Oscar

Update Feb 28th, 5:00 pm PST: Backups have been mounted to the new machine and the apache instances are starting up now. Once all apache instances are up customer sites will be up and running off the backups. Backups are mounted via NFS so that your sites are up while the restore runs. Once your website is up, if you notice missing or outdated data, or experience any other issues, please be sure to contact support.  We will update this post again once all apache instances are up. – Oscar

This entry was posted on Tuesday, February 28th, 2012 at 3:12 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

37 Responses to “Shared web server ‘mystique’ moved to new hardware and restoring from backups (RESOLVED)”

Its nice you guys are working on restoring this server, but when are you going to restore Stampeders? Would like to get my sites restored.

thanks!

As of this morning I am not receiving email and my website is down. Please contact me at kt6319@gmail.com

Machine “microscopium” is not working and I had clients all day getting really angry at me. Please try to resolve this, its been all day like this.

Oscar, we are now into our FOURTH day of downtime, that’s four days of lost sales and damaged credibility. Justin may have addressed our tickets personally but he also said he had sent us an email which we never got. I assume this email instructed us on how to upload our new sites to a new server?!?

I cannot believe one company can screw things up for so many people, you people are single handedly destroying e-commerce!

The site was up for some time in the morning (India) but again went burst. Surprized that how things can take so much of time. It looks that its a complete break-down.

Pls let us know by when you presume the sites to be up and running

Santosh

I just don’t understand why you keep having continuous outages. You have to do something to retain your customers. I know that every hosting company has outages, but this has been one after another. Not expecting perfection. And you do have great customer support. But the service itself is lacking big time.

Google:

Your search – RAID site:status.dreamhost.com – did not match any documents.

Heh. It’s no wonder you stop Google indexing this site. LOL.

Still getting RuntimeError (!!! Missing the mysql2 gem. Add it to your Gemfile: gem ‘mysql2′) errors
for my Rails site. Gem list and bundle gem both show mysql2 is installed. This site had been working for over two months before yesterday’s outage and restore. Filed a ticket but no response yet.

Sure would like to get my site back up.

One hour after “All sites should be up and running as well”, I’m still down…

I need help Ps6227 down.
48 hours and my server is dropped.

@Jon: Stampeders should be in better shape now, I’ve replied to your tickets and made sure your site was working.

@Kirsten: We cannot give details about customer accounts to emails that are not the primary contact email. We have written the account owner about this domain on your behalf.

@Gaston: microscopium has a 12 day uptime and normal load. Your site is loading and your ticket responded to.

@Mark: The ticket reply was sent, it would go to the accounts primary contact email. I am working on your two domains that were moved right now, and will write you again shortly.

@Justin .. can you please check out ps37457?

I have been following that thread for 3 days now and every time someone posts that it is fixed it doesn’t work. I understand you guys are seriously swamped, can you at least reply to one of my support tickets or posts? I need to know when we can access data at the very least. The blog posts say to contact support, which we have done. It looks like you are ignoring people and only responding to others. I am the primary account holder on the VPS. Please advise!

@Santosh: The site is loading now, the VPS restore for homie-vserver19 is continuing.

@BT: We are aware it is a larger number of issues at once than normal, there are plans for improving services to the customers and will be slowly rolled out in the future.

@jacoulter: Ok, looking at your issue with mystique.

@Rob: The server is working at this time, and you didn’t provide any details for me to check a domain or account.

@david: ps6227 is currently running, with a 8 hour uptime. I see data may not be coming from the backups though, checking it.

@weknowsnow: Sure, I’ll check out your issue.

@Justin thank you and best of luck! Looking forward to your reply.

Justin, Can I please have an update on ps38606? I assume it’s running on old backups, because half my data (photos) is missing. I really need to know if this data is lost for good, or if it’s eventually going to be restored. It is so frustrating to see others getting help, and yet I’ve been left in the dark for four days now.

I am spending money to advertise on facebook for my site, who I shall repay you??
and when you make it all work properly!

ps7997, I lost ad revenue for 3 days now and people are angry. I’ve been with you guys for 7 years and I can’t even get a response. I know you’re working hard but we deserve more of an update. TO put it bluntly, Our lives are in your hands and you have butterfingers????!!!

Hello
Is there any updates as you are not responding to any tickets?

I cannot access my email and it is impossible to work. My inbox is missing and I cannot even login on the web browser.

I am trying to find help since my replies go unanswered. Apparently my server (a PS) is up but has ZERO files. Dan told me this: “All sites should come up with the data that we had in our backups prior to the server crashing. Contact support if you need specific information.”

Does this mean, my files were not backed up properly by dreamhost? and I am now screwed? I had 30 client sites on there…..

Justin: I have a rails site and all of the files appear to be there, but my semi-educated guess is that the phusion-passenger wasn’t restarted

Exception class:
PhusionPassenger::UnknownError

Hi,

Lots of my sites are down and all are on barker, not the machine listed. Any update on what’s causing these issues?

cookingisfun.ie
toflidium.com
jobsforcooks.com

Same problem here. My website on Barker is dead. I can’t access to the http://FTP... Help please !

Please could you update status to indicate that Barker is down. It seems a bit silly to have a status page that doesnt reflect status!!

My Rails 3.0 site (which ran without any issues for several months before Mystique’s outage) is still down – complaining the mysql2 gem is missing but gem list shows mysql2 0.3.7 installed locally.

Tried adding ‘gem mysql2′ to my Gemfile and restarting Passenger but no luck.

12 hours after opening a support ticket I received an auto mail informing me it had been moved on to a support person’s queue. It was moved on to the queue over 18 hours ago.

That must be some queue.

So I’ve been down for almost three days. Dreamhost is a cheap solution for testing and hobby sites, but after this and reading about all the recent VPS outages I couldn’t recommend them for a production site.

I have new dedicated server ds3152 and after 24 hours all my sites are still down! After the events of the past month my clients are ready to jump ship. Why is nothing working?

@Desperate Zombie & @Toby: barker has a 15 day uptime as well as a 15 day uptime for the random apache I checked. I don’t see any issues with this server at this time.

@rob & @jacoulter: I updated the status post with info about rails and upgrades needed, because it appears the operating system version changed when the server was replaced, and the newer rails version has some changes compared to old.

@Christine: It appears that your issue has already been addressed, your last ticket will be replied to soon.

alora!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks Justin, back up now

I NEED HELP!!!!!!!! lokk what you say to me:
“Hi Daniel,

I apologize, I’ve gone through all of our backup servers, and have been
unable to find backups for any of these users. While we do our best to
maintain backups for all users, we can’t guarantee their availability,
and you’ll need to restore these from your own backups. I’m very sorry
that there’s nothing more we can do for you in this matter.

Thanks,
Brandon”

Hey guys, don´t say you don´t have backup, in the post about michaelkelly you said you had the info from the old hardware, well, there has to be the information, the sites that are online, is because i reuploaded them, I NEED THE INFORMATION TO BE AVAILABLE; I HAD SVN, I HAD MORE THAN 130GB THAT NEED TO BE RESTORED, THIS IS NOT MY FAULT!!!! I MOVED TO THE VPS TRUSTING YOU TO HAVE THE OLD HARDWARE BACKED UP, I NEED THESE USERS URGENTLY!!! I DON´T KNOW WHAT YOU HAVE TO DO, BUT I DON´T HAVE BACKUP OF THESE USERS, AND THEY ARE THE MOST IMPORTANT ONES!!!! PLEASE SEND THIS TO SUPERVISOR, OR MAKE MAGIC, I DON´T KNOW, IT´S SIMPLY IMPOSSIBLE YOU DON´T HAVE THE INFORMATION IF YOU HAVE THE OLD HARDWARE, I NEED THIS TO BE FIXED AS A MATTER OF DEATH, MY CUSTOMERS ARE KILLING ME!!! A LAWYER CALLED ME THIS MORNING, PLEASE HELP!!!!!!!!!!!!!

davellanedam
intermarket
parperf
queenmaster
totumo
chiky_fly

@Daniel: We do not have the data you requested. Backups should be made for all important data off-site.

@rob: Glad to hear it is back up now.

I’ve put a ticket in to tell them to cancel my account but guess what………….they are not replying to that either.

@Justin I´m sorry, but YOU said this “Update Feb 28th, 9:45am PST: We believe the restore is complete, but want to have admins verify it just to be sure before resolving this issue as completed. All data from the backups and from the old hardware should be copied over, if not, please contact support detailing what you are missing and we can restore it for you. Please be sure to Contact Support if you have any questions or concerns. – Justin”

Hi guys, i´m dying here, i just logged into michaelkelly with some usernames and the info is there, i just need you to move from michaelkelly to my VPS ps86583, i know you already did it, but when i requested a move you were in trouble, and i´m seeing some restored info there, so, please, MOVE THEM AGAIN!!

@Daniel: I replied to your email already and we already restored your backups.

@Danny: What’s your webid and I can help you with closing your account.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups (RESOLVED)
2:37 pm

Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 27th, 2012 at 2:37 pm PST) by Brandon

Due to complications with the drives on the web VPS host server ‘homie-vserver19′, we are moving the server over to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this situation. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact support with the details.

Please note, this is regarding the web VPS host server ‘homie-vserver19′ and no other services (email, MySQL) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number. -Brandon

Update Mar 5th, 5:15pm PST:We’re happy to report that this incident is now resolved. All guests are restored, but if you’re missing any data or are having issues with your sites, please don’t hesitate to contact support with the details. – Andrea

Update Mar 5th, 4:15pm PST:All VPS guests have been restored, we’re just waiting for some data sync before we can resolve this status post. Everything should be working as expected. Thank you for your patience in this process. – Andrea

Update Mar 4th, 9:05pm PST:We only have one more guest (ps8041) to be restored on this server before we can resolve this incident. We appreciate your patience during this process. All VPS guests should be up and running according to our records. Please contact support if you need any assistance. – Andrea

Update Mar 4th, 4:00pm PST:The restore is down to the last VPS guest to restore, which is the biggest in disk usage, so the restore is almost done. All VPS are running at this time, if you are having any issues please contact support. – dan w

Update Mar 4th, 8:20am PST:The restore is down to the last two VPS guests to restore, which also are the biggest in disk usage, so the restore is getting pretty close. All VPS are running at this time, if you are having any issues please contact support. – Justin

Update Mar 4th, 2:10am PST: We’re definitely getting closer to the completion of this restore process. We have two VPS guest machines still restoring but all servers are reporting as online at this moment, if you are having any issues please contact support. — JJ G

Update Mar 3rd, 8:10pm PST: This restore is still running without any issues and all VPS guests are online. The restore is getting close to finished, so we’ll keep you updated. If you experience any issues with your sites or are missing data, please contact support for assistance! — Brian H

Update Mar 3rd, 1:35pm PST: This restore is running without any issues and all VPS guests are online.  If you experience any issues with your sites or are missing data, please contact support for assistance! -Oscar

Update Mar 3rd, 5:55am PST: VPS services on this guest remain up and the restore process continues to execute. If you are running into any problems or if you have any questions, about a VPS on this machine, please let support know! -JJ G

Update Mar 2nd, 10:00pm PST: The restore process doesn’t appear to be having any issues at the moment. All VPS look to be online. If you experience any issues with your VPS, please contact support.  -Dan J

Update Mar 2nd, 4:32pm PST: The restore process doesn’t appear to be having any issues at the moment. All VPS look to be online. If you experience any issues with your VPS, please contact support.  -Dan J

Update Mar 2nd, 11:20am PST: This restore is running without any issues and all VPS guests are online.  If you experience any issues with your sites or are missing data, please contact support for assistance! -Oscar

Update Mar 2nd, 4:57am PST: This restore is continuing with no reported issues at this time. All VPS guests are online, but if you experience any issues, please contact support for assistance! -Patrick

Update Mar 2nd, 12:55am PST: This restore is continuing with no reported issues at this time. All VPS guests are online, but if you experience any issues, please contact support for assistance! -Patrick

Update Mar 1st, 5:40pm PST: The restore is continuing smoothly on this server and all VPS’s are up and running. If you experience any issues with your VPS, please contact support. More updates to come as the restore process completes! – Justin

Update Mar 1st, 10:00am PST: The VPS are all up and running and the restore is continuing. If you experience any issues with your VPS, please contact support. More updates to come as the restore process completes! – Justin

Update Mar 1st, 3:40am PST: Currently, all but 3 VPS guests are having issues, which are being looked into by our admin team. The restore is continuing smoothly, and we will update with further news soon. Thanks again for your patience! – Chalise

Update Feb 29th, 6:30pm PST: All but 2 VPS guests are up and running but the restore is continuing. There are currently some Apaches reported down, which is being looked into by our admin team. We appreciate your continued patience.  We will continue to provide you with updates so please check back again soon. – Seohee

Update Feb 29th, 9:15am PST: All VPS guests are up and running and the restore is continuing. If you experience any issues with your VPS, please contact support. More updates to come as the restore process completes! – Justin

Update Feb 29th, 4:10am PST: All VPS guests are up and running, and the restore has been started up again, and is running smoothly. If you experience any issues with your VPS, please contact support. More updates to come as the restore process completes! – Chalise

Update Feb 28th, 7:40pm PST: The backup brick has run into issues, which means the restore is halted and all VPS guests are down at this time It is being looked into now, and we’ll make sure to update this post as soon as we have further details. We apologize for the inconvenience and appreciate your continued patience. – Seohee

Update Feb 28th, 5:15pm PST: All VPS guest machines are currently running and we are working to fix any guests reporting issues with their apache instances . The restore is progressing without any issues. If you experience any issues with your VPS, please contact support. More updates to follow as this completes. – Oscar

Update Feb 28th, 8:30am PST: All VPS are currently running, but we see some that are reporting their apaches down. We are investigating the causes of this to get all services working during the restore process. If you experience any issues with your VPS, please contact support. More updates to follow as this completes. – Justin

Update Feb 28th, 5:15am PST: The restore process continues to run smoothly, however we do see a few guests that will need to be manually restored by our admins, which they are currently working on. If you experience any issues with your VPS, please contact support. More updates to follow as this completes. – Chalise

Update Feb 27th 9:53pm PST: All VPS should now be running again mounting the backups. The restore process is going as well. If your private server is not running or then please contact support with details. We will continue to update this post as the restore process continues. – Dan J

Update Feb 27th 8:53pm PST: We’ve run into file system errors from the backup brick. We’re working on fixing this up and getting it mounted on the new hardware, then plan to continue the restore. We apologize for the convenience, and will provide you with updates as soon as we have them. – Seohee

Update Feb 27th 5:15pm PST: The new hardware is ready to go, and we’re setting the mount so we can serve your sites through backups during the restore. We’re starting the VPS guests one by one. At the same time, we initiated a file system check on the old hardware, and if it checks out OK, we can revert the move and stop the restore from backups. The file system check will take several hours, but that is now running. We’ll update this post again soon with more information. – Andrea

Update Feb 27th 3:21pm PST: We are currently working on setting up the new hardware to which we’ll be restoring the affected VPS guests. As soon as the new hardware is ready, we’ll be able to serve your sites again from backup and use that same backup to restore data to the new hardware. We  will  keep this post up to date with new information as it becomes  available. -Brandon

This entry was posted on Monday, February 27th, 2012 at 2:37 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

145 Responses to “Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups (RESOLVED)”

List of affected VPS guests:

ps26803 ps27038 ps28747 ps29515 ps29587 ps50445 ps55596 ps55830 ps56793 ps56889 ps5787 ps5793 ps5796 ps5800 ps5804 ps5810 ps5813 ps5835 ps5905 ps6187 ps6193 ps6210 ps6227 ps6328 ps6341 ps6352 ps6472 ps6486 ps69372 ps69405 ps70335 ps7362 ps7370 ps7376 ps7719 ps7721 ps7997 ps8041 ps8257 ps8270 s###pants

Guys – everything I’m currently hosting for clients is down. Can you please advise? I don’t think I’m on that host.

how can we move back to shared hosting until you have this fixed. we moved from shared to VPS recently. please reply asap

Hey guys,
Thanks for the heads up. Any ETA on the new hardware?

Also, I’m just curious as to why 7 of the last 11 DHStatus updates have been virtually all the same problem, but different servers. Why does your hardware keep failing? Sounds to me like there is an underlying issue that may be getting overlooked.

Not on that list of affected VPS but our site has been either really slow or down all day today, AGAIN.

We’re not on that list of affected VPSs. However, our site has been either really slow or down all day today. Can you please explain why? Submitting ANOTHER ticket now.

Could you please tell me if ps75586 is affected? All 45 sites are down!

I’m wondering the same thing Marty. I have the RSS for the status page in Google Reader, and I’ve kind of been expecting mine to eventually get caught in the mess eventually. My hope is that the migration fully resolves whatever issue has been affecting so many of the VPS servers and that I won’t have to worry about it after mine comes back up from this outage. I’m keeping my fingers crossed.

This is ridiculous. I am on PS41422 and this server is not listed above. I host for several clients who are now pulling their service from me.

@Jennifer – I checked your site, severien.com which is on a shared machine and is loading fine. If you need further assistance, please contact support through the DreamHost panel with the specifics to your case.
@ogp – we have contacted you
@Marty and Atnevon – the new hardware is up, and the VPS guests are being brought online. As for the numerous cases, it’s just an unfortunate coincidence. We’re also looking into the old hardware through, to make sure it’s fixed.
@Sergei – all affected machines are listed in comment #1, your VPS is on a different host. The machine was offline, I’m rebooting it. Your sites are online again. You can also use the server reboot option in the DreamHost panel: https://panel.dreamhost.com/index.cgi?tree=vserver.reboot&
@Andrea – Please refer to http://www.dreamhoststatus.com/2012/02/25/web-vps-host-server-%e2%80%98homie-vserver173%e2%80%b2-moving-to-new-hardware-and-restoring-from-backups/ for updates.

slutpants? Really?

More problems with the VPS. How can I move back to Shared Hosting?

My site is down now. What’s the eta for it to be back up? I JUST launched a few days ago – this is really frustrating as I’ve had people contact me to tell me that my site isn’t working. Couldn’t be worse timing!

My site is down as well. On another VPS server than ones listed above. When we expect service to be restored. We have client deadlines for tomorrow!

Pings are dying at router-0.hq.newdream.net – VPS network issues now?

This is so frustrating! trying to launch a simple brochure site and i can not even do that!

Please help, support tickets take forever.

brumbys shared is giving me this. Randomly works and then not,

Database Error: Unable to connect to the database:Could not connect to MySQL

We are having issues today. Is this still affecting VPS’? Are others down?

Ok, so after being down for a day on Sunday when my PS wasn’t listed on any of the various problem machines, now we are listed

Ok, so after being down for a day on Sunday when my PS wasn’t listed on any of the various problem machines, now we are listed – ps5796 – so we’re down again. Can somebody press some kind of button to get this s**t sorted please?

Hello,

We are on PS40039, we are facing issue but even it not listed anywhere in this post.

Regards
Mahendra

Is there any update on this at all, this is the 4th time in 3 weeks.

Hello on ps30342 which is also not responding, though not on the list

VPS down: ps6227
Server down again in a few weeks. I have over 100 websites and over 40 fall advertising campaign stops and thousands of dollars lost again ..

Website down yet again. 4th time in 3 1/2 weeks.

this is awful. PLEASE JUST GET ME ACCESS TO MY DATA.

I’m not sure exactly what’s going on, but it appears to me that I have an incomplete restore. Random images and segments of PHP appear to be missing from at least one of my sites. Things are functional, but it looks messy. I can do a restore from my backups, but I’ll wait until things are fully complete before I start, as I’m not sure if maybe things are just still in the process of restoring.

@Dreamhost: Can you pls explain why you guys are so inefficient. You can’t fix your down servers for last 12 hours; Just ridiculous.

Correction to my last post: It turns out that my ‘cache’ directory of my Invision forum software was empty. It was a little tricky getting that back, as it contained a language cache as well (so I couldn’t really see what I was doing), but a little recaching seems to have fixed everything I can see so far.

i have sent 4 support requests over the last day for my VPS. all ignored. why would you post on every comment to contact support if you ignore it? your blog/status posts said my system should be up as of yesterday. still doesn’t work, you even replied to the ticket saying all was good. i have rebooted multiple times, no dice. what do we have to do to get support and/or correct information?

I agree. WE NEED ACCESS TO OUR FILES ASAP. I have multiple clients who have been down since Sat night. This is horrible business and I’m losing money and clients each day – because they are losing money.

is my vps ps5905 complete the restore process? I can ping to my domain but cannot log on to my ftp.

My VPS is not working. How long is that going to take?

ps25602 offline – second day. Two tickets in and no response. Third time offline for 48 hours or more in a month. This is simply unfair.

Can you please atleast have our sites on some temporary machines while you are moving?

ffs all my sites gone again having just spent another day fixing the mess you made the other day. Come on fix the f—ing things, you give the impression your a bunch of kids f—ing about in someones spare room.

Is it possible that we atleast have a backup of our data? I can have the server temporarily available for staff at my own location.

Can somebody please call me – urgently? I need just the backup file for mysql.

This is totally ridiculous, no reply to tickets and not even any updates on here, I wonder if Dreamhost is going out of buisness??

ps23862 down for a couple of hours. Not listed on any of these updates.

Please don’t tell me I’m in for another week of terrible access :(

Justin are you there? You were the magic man last time. Please help with the data backup.

ps37457 is having problems. I have contacted you via support, no reply. VPS is down

I’m just stunned that these problems keep arising as I’ve had issues with Dreamhost SHARED hosting for the past 3 months continually, after 10 YEARS of relatively decent service. The RIGHT thing to do Dreamhost, because you cannot seem to keep servers up or current backups running (my site shows so much outdated info or missing images its not funny), the ability to sftp, OR just ftp …. the right thing to do is give all of your clients whom you have failed over the past many weeks a few free months of hosting. Is that too much to ask? It will never make up for lost business, lost clients, or US looking like idiots to those who visit our sites that arent working right (if they load at all)… but for the love of good business, extend some sort of token of apology because quite frankly, the uptime you happily advertise is now completely false.
Regards, someone who used to be a very HAPPY Dreamhost customer. Looking forward to another real cutsie newsletter that wont even casually mention all the BS that has been going on!

I’m on ps38606 and none of my images are loading (IF the site even loads). This has been going on for 24 hours plus now, and not a single response to my three support tickets. Frustrated is an understatement.

@ Lori I couldn’t agree more. With DH since 2005 and have always been happy, both with uptime and with customer service, but this year has been completely ridiculous. We have lost clients, lost brand goodwill, and pissed off a whole load of people. Just saying ‘sorry’ isn’t going to swing it this time. No longer a happy Dreamhost customer.

The backup for http://www.expenews.com you charged in ps25733 is a old version of our applications from TWO MONTS AGO!!!

We have had three sites down since Sunday, Ive stopped thinking about the money we have lost as it just annoys me. To top it all when our IT dept tried to get someone to call us (a service we pay for) we were told US customers take priority (we are UK)and no one would be calling us as we were non US!. So it seems DH have a scale of customer importance, but having read the comments on here it seems its US customers dont feature particularly highly on the agenda either.

What is going on?

Sorry for the delays, we are working to get all issues resolved as soon as possible.

@Danny: Your vps is now up and mounting the backups. No, we are not going out of business.

@Ritu Sharma: Your ticket was just answered by Miguel. If you have further issues please let us know.

@Craig Atkinson: You are not on this vps host machine, you are on stampeders, which has its own status post. We are aware of the issues with the entire machine and are working to get the backup server behaving.

@Nathan: You are not on this vps host machine. We are not experiencing any issues with homie-vserver158 right now, which is your VPS host machine. Your site is loading, I’ll have your tickets answered up shortly.

I need to know when the sites will be back up and running. Both sites have been down for 24 hours!

Hello Dreamhost team, my ps5905 is not up yet. May I know how long will it take for the whole cycle? Will there be any chance that I will lost my data?

It will be better if DH give us advance notice for such migration so that we can make our won arrangement for data backup and necessary announcement to our web users. We also would like to request for more comprehensive and transparent inflammations and regular frequent update during the process. Hope to hear from your response, Sirs.

2 days (so far) to migrate to new hardware? If I had access to my files, I could migrate to another host in less than 5 minutes. I also think some refunds are in order.

Please help me. I am an amateur webmaster doing a volunteer job of supporting our homeoweners web site http://www.meadowcrestvillages.org. I have been doing this for several years with only a few problems — last year when you moved to a new server and today. I tried to update the remote ftp and I keep getting a message that the server does not exist. Is this because of a server that is down? Moreover, even though we subsribe to your service I do not know how to access the DreamHost Panel as I do not have a password. How did that happen? Would appreciate some help as 600+ homeowners are awaiting the changes to our website. I’m in a pickle as you can see.

@Dave Could you please provide us with your VPS guest ( psXXXXX )?

@Charlie Your VPS is back online. Also, the filesystem completed on the old hawrdware. Therefor, there is no data loss. If you are missing anything, please submit a support request and provide the full url to the missing data.

Lastly, The migration to new hardware was not planned. Our admin noticed that a few drives were faulty and felt that the best course of action would be to move the host machine to new hardware.

@John Transferring TBs of data does take quite a bit time. Sorry for any inconvenience this has caused. We truly appreciate your continued patience.

@Sharon Ziemba: We aren’t having any issues with your webserver, if you are having trouble to ftp, it is likely a wrong server name or hosting in the ftp program. I would check that it is the domain name, and not a server name (servername.dreamhost.com) or other odd setting.

@John: It can take up to a couple weeks to migrate to new hardware, because the old hardware is often not available, and the 1TB to 4TB of data can take some time to copy while the webserver is live, from the backups.

@Charlie: ps5905 is running but nginx isn’t. Looking at it right now.

Brian

We have sent two tickets today, no response. Been told that US customers only are allowed phone support, disgraceful. Posted several messages on the various update blogs in the last few days, been ignored and finally OUR SITES HAVE BEEN DOWN SINCE SUNDAY what the hell is going on???

and before you ask our VPS is ps39975

@Mark: Looking for your tickets, please standby.

Hay!!! my server PS5905 is up and running now!!! Thanks you so much Dream Host team. May you all be happy, wealthy and success in your hosting business.

ps27109 down too, no response to two support tickets lodged, two days and counting…

Justin, so you are saying that it could take a couple
of weeks for some sites to be restored? Is that correct?

@Mark: Still looking at your VPS. No, the site should be up and running off of backups. I was referring to the amount of time it will take to restore all data from backups to the new hardware.

@ken: Still looking at your VPS.

Brian. My VPS Guest is ps37457. Thanks for looking into this.

dave

So if the site I’m using from a friend is still down what does that mean?

It went down when all of this crap started and I’ve waited patiently for everything to be up.

Now that the message is that everything is up but that site isn’t….what does it mean? :o

Web Server PS55830 still offline.

Justin, we now have “site temporarily unavailable” on two of the sites and server can’t be found on the third.

Me again – I see various people seem to have been getting replies to their comments on here from DH people. We, on the other hand, have had nothing, no replies to support tickets, no comments on here. We were down all day Sunday, up for about 24 hours then down again for the last 24 hours. This VPS is part of a business structure on which a number of paying clients depend, including our own company website. Any chance of you paying some attention here? ps5796

I have also submitted several support tickets with no direct response, and no response here in the comments. This is an absolute nightmare from a business perspective. Justin – can you please give me an update on ps38606. It was down, then it was up (but not loading correctly) and now it’s gone again. I really need answers!

@Justin: Yes, the site SHOULD be running off backups. However, ps5800 is not, and hasn’t for 2 days.

Our server is still down.. ps63949. WTF this is the longest downtime ive ever seen in the history of all web hosting providers!

ps54088 still down, 2 long outages in less than 30 days.

ps6227 down

I lost 4 customers because they are tired of having your web fall. I have 20 hours to receive customer complaints. I lost a lot of money. This is not professional. Many businesses rely on 100% of their sites to work. I have it clear, I take my three servers and all the domains to another company. Desire not to suffer a day like today. And certainly after 12 hours I have received no response.

Would someone from DH staff please take a look at our VPS? ps37457
It hasn’t worked for over a day now!

@dave: Checking out your vps.

@don: That isn’t enough information for me to check anything. Please provide a domain name, open ticket number, or something that I can look up.

@Ken: Should be fixed.

@Mark: I emailed you the details, that will show for another few hours while the domains dns update to the new vps they were moved to.

@Mike & @Emily: We are doing all we can to get all affected people back up. I will take a look at your VPS next to see what I can find.

@John: Fixing yours up right now.

ps7997 still down. please help

What about ps55484? Still down since Sat!

Justin, we have received no emails from you and I don’t understand your message, what will show for another few hours?

I feel as though we are having to battle tooth and nail for information and for a service that we are paying you to provide. You are dealing with peoples livelihoods here!

I see, you mean the “temporary unavailable” will show for a few hours! Are you saying after that all our sites will return? I shall hold you to that

Any updates on ps5813?

All of my sites are still down and have been since Sunday.

Justin, we have no emails, please resend to

Info@thehistorybunker.co.uk

Ps55830 Site Down… When I SSH I get a log in prompt but user/pass does not work.

Ps55830 – When I SSH I get a log in prompt but user/pass does not work.

Please see the server Ps6227. Its more important.

ps54088 down, please help.

ps7370 still down. Now greater than 24 hours. My business and those of others have multiple sites that run. This is the second time in the last six weeks where I have greater than 24 hours without any service.

Like others – no response to multiple help tickets. no updates. no warnings. no answers.

I’m on PS46554 and am down as well.
I won’t add to the rage expressions though I share them all.

I am currently losing a very big contract because you’re unable to serve my sites. I am really really disappointed.

ps7997 is down for more then a day now. PLEASE give us an update.

@John: Your VPS is up and running off mounted backups.

@Chris: Your VPS is not hosted on this machine, it’s on hv173. I’ll be checking on it soon, but please post on that thread.

@Mark: Your VPS is on hv173. Justin already addressed your tickets personally and your sites should be up. DNS propagation can take anywhere from 4-12 hours so you’re going to need to give it some more time to update if your sites are not loading fo ryou.

@Daelan: Your VPS is up and running off mounted backups.

@Patrick: Your VPS is up and running off mounted backups.

@Steban: Your VPS is hosted on hv142. I’ll be checking on it, but please post to the correct thread.

@Scott: Your VPS is up and running off mounted backups.

@Mushon: Your VPS is not hosted on this machine, it’s on hv173. I’ll be checking on it soon, but please post on that thread.

@Oscar: what about ps37457? People have posted they are working on it a few times but never any follow up?

ps7997 is barely running, the sites are taking forever to load. Please don’t say things are working when they arent

I see the update that sites should all be up, but mine is still down. ps38606. I know you guys are working hard, and I’m sure you’ve all had a really bad day. I appreciate that you are trying to resolve the issue. I think the majority of the anger here really comes from the lack of communication, not from the outage itself. Please (PLEASE) let us know what’s going on, and at least when we can expect to see our sites up and running again. We all have customers we need to communicate with as well.

ps54088 still down. Please dreamhost help, my clients complain and I loose sale’s and money. This is the second time in less than 30 days.

Hello Dream Host team,

My VPS server went down again. It was working for 6 hours. Please help to check my server ps5905.

Thank you for your hard work!

Our apologies for the downtime. If your VPS is on HV19, please review the most recent update. We are working on getting this back up and running as fast as we can. We appreciate your continued patience.

Bluntly put and to add to the numerous reports of VPSs still being down (when you’re reporting all are back up), be advised ps29785 is still down and this is unacceptable. It’s been down for over 30 hours.

Please look into immediately and advise. We’ve also tried to reach you via Twitter to no avail.

Guys – everything I’m currently hosting for clients is down. Can you please advise? I don’t think I’m on that host.

And it has down for more than 2 weeks.
I have reboot and point dns to DreamHost.

Oscar, we are now into our FOURTH day of downtime, that’s four days of lost sales and damaged credibility. Justin may have addressed our tickets personally but he also said he had sent us an email which we never got. I assume this email instructed us on how to upload our new sites to a new server?!?

I cannot believe one company can screw things up for so many people, you people are single handedly destroying e-commerce!

Please review the Ps6227.

Google is penalizing us for having the websites unavailable. I’m losing months of work. It is a RUIN my business. Please fix it quickly.

Ps6227

Please solution the problem with this server. My business depends on it. I am losing a very significant amount of money due to the fall of all webs. Several customers have been angry enough and I lost. It is tremendous damage to my business. Please take priority in the review of my server

Clearly you know about this already, and you’ll have read all of everyone’s comments and support tickets about the level of service being totally unacceptable. Which it is, and I hope that something dramatic is done about this otherwise we are out of the door and I suspect a lot of other people will be too. But I need to make a point about customer service – which when you had Live Chat and you actually responded to support tickets, was actually the main reason I have stayed with DH for so long in spite of occasional hiccups in service levels.

The way this has been handled – no replies to support tickets, intermittent and uninformative posts on DHstatus – makes it feel like there is only one person at a time working on the problem, and that that person is both trying to fix the issues and trying to reply to your customers. A decent-sized company like DH surely has a CS department and a huge tech department (since what else do you do for a living?) – and if so, the hilariously slow response time and the absurd inability to handle the problem make us – or certainly me – feel like we are a tiny blip about which you don’t really give a sh*t. People’s whole businesses rely on their websites – I’m sure we’re not alone in having office backends run from here too, so a number of clients have no access to any of their office information or tools and might as well take the next few days off – except that they have clients, and their clients are pissed off too. As an example, one Tour Operator whose office runs off this server has no idea where its clients are – while they are in resort – nor does it know when they are leaving, or when the next lot are arriving, or even who the next lot are. Another client was supposed to be making a pitch yesterday/today to a major corporation for over $100k of development funding – but they have nothing to show, since they have no website. I could go on. I am not making this up. I am sure we are not the only company in this position.

The lack of communication is ridiculous. We – and plenty of others here by the looks of things – have a lot of angry customers, and those customers have questions:

1. When will it be back up
2. When will it be back up
3. What happened
4. Why did this happen
5. How do I know it won’t happen again
6. What am I paying you for
7. When will it be back up
8. When will it be back up
9. etc

And we can’t answer them, because we don’t have any contact channel to DH and no one is telling us anything except “we’re working on it”.

Also, it feels to me as though you’re working on Pacific time and out of hours not much is happening at all. But you have an international clientbase – we are in France, for instance, and we need service during office hours here. If you are working through the night on this – as you certainly should be – then again, you need to sort your communication out.

I’m really f’ing angry, but more so I’m embarrassed, and my company is embarrassed. Any number of businesses are going to be very significantly hit by this, and as far as they are concerned it is our fault.

Ps6227 down.

40 hours and my server is dropped. It’s a disaster. My business is sinking.

This is truly absurd. I’m glad my time with Dreamhost is up for renewal in two weeks. It’s too bad as I’ve been with them since 2006 and for the most part things have been fine – it’s been a recent problem only in the past year or so. I’m definitely looking into switching.

Anyone ever try: http://www.webhostinghub.com/ ? I can get 3 years of hosted space for the same price as one year of Dreamhost’s “99.9% Uptime”. Or does anyone have a different recommendation for similarly priced hosting?

Ps6227 down.
41 hours and my server is dropped.

¿Tengo alguna alternativa o me aconsejan que me vea poco a poco como me arruino? Contraté VPS de Dreamhost porque buscaba seguridad.

Do I have any alternative or advise me to see me as I gradually ruined? Dreamhost VPS hired because he sought safety.

@Mike

Very well put indeed, my sentiments exactly, I especially agree with you on the number of people (or lack of) working on the problem and the international time zone situation.

DH, you are ruining peoples businesses and livelihoods, do you care???

Ps6227 down.
43 hours and my server is dropped.
Do I have any alternative or advise me to see me as I gradually ruined?

Ps6227 Down

Incorrect:
“Update Feb 29th, 4:10am PST: All VPS guests are up and running, and the restore has been started up again, and is running smoothly. If you experience any issues with your VPS, please contact support. More updates to come as the restore process completes! – Chalise”

Ps 38606 still not working correctly…

Ps6227 down.
45 hours and my server is dropped.

ps5813 is still down.

I’m now losing work because of this.

February 29th, 2012 at 6:10 amAlexander Kjerulf Says:

My site at arbejdsglaedenu.dk is back up but only showing an error me’tsagetth6at wasn’t there before. Is that being looked at?

February 29th, 2012 at 6:12 amAlexander Kjerulf Says:

My other sites are up again but arbejdsglaedenu.dk only shows an error message. Is that being looked at?

Ps6227 down.
47 hours and my server is dropped.

Ps6227 down.
48 hours and my server is dropped.

ps7997, I lost ad revenue for 3 days now and people are angry. I’ve been with you guys for 7 years and I can’t even get a response. I know you’re working hard but we deserve more of an update. TO put it bluntly, Our lives are in your hands and you have butterfingers????!!!

@ Omar Tapia: Your VPS is up and running off mounted backups. Your site is loading fine for me now.

@ Mark: your server is on hv173, not hv19 listed for this post. Make sure to post comments in that post. Justin responded to your support requests. If you didn’t receive them check the ‘support’->’support history’ section of your panel.

@ david: Your VPS is up and running off mounted backups. I tested several of your sites and they are all working.

@ emily: Your VPS is up and running off mounted backups. I tested your sites and they are all working.

@ daelan: Your files were not restored in the correct location. I fixed that now and verified your sites are displaying now.

@ alexander: your server is on hv142, not hv19. Make sure to comment in that post. Your VPS seems to be up and running and service sites though.

@ john: Your VPS is up and running off mounted backups. I tested several of your sites and they are all working.

ps7370 still down

Any updates regarding restore? Tickets unanswered and live chat has been disabled during this event.

DAN , my sites arent working. There are no files in any of my folders!

Hi Dreamhost,

My VPS ps5905 is up and running now. But, MY DATA are been wiped out. The entire website is empty now. I have 900GB of data on my PS5905. HAVE YOU DELETED ALL MY DATA?

my vps5905 is empty. When I ftp to my vp5905, there is nothing in my folder. I was told previously that file system for vp5905 is completed. It wsas runing for 5 hours and down since yesterday. Now, I can login to my vp5905 ftp but there is no data in my ftp. I am concern that I lost my data. Is My DATA still there? It puzzle me that my vp5905 data restore process is completed and running for 5 hours with complete data and now the entire vp5905 is empty. Does it mean I lost all my data??????????

I would like to know Dream Host’s SLA (Service Level Agreement) commitment to customer. is it 99.99%? Does DH has proper migration plan and Disaster recovery plan? Hardisk failure and any other hard failure is norm to hosting industry but it shall not effect the operation and it shall not affect the system down time.

I pay 200 a month for my PS. and now all my files are gone.

Hi DreamHost Team,

Now, my ps5905 is up and running again. My data are still there.

Thanks.

March 1st, 2012 at 2:21 amAndres Gonzalez Says:

I totally agree with charlie. We’re in the same situation: no files, no previous notification and (as far as I can see) no backup. I’d like to have a look to the SLA document. But first of all… restore my files!!

Andrés

p7370- We deserve an update please

@Scott: I just replied to you, your VPS should be up now.

@Sune: The site appears to be loading at this time, if you still have issues please let me know.

@John: Replied to you.

@Charlie: Glad to hear your data got restored. Please let us know if you see any missing data.

@Andres Gonzalez: I’ll see if I can look up your account by name and will be in contact with you.

@Daniel: You sent this to all blog posts, but it doesn’t change the situation. We do not have the data you requested in a backup, please restore from your own backup.

@Justin I´m sorry, but YOU said this “Update Feb 28th, 9:45am PST: We believe the restore is complete, but want to have admins verify it just to be sure before resolving this issue as completed. All data from the backups and from the old hardware should be copied over, if not, please contact support detailing what you are missing and we can restore it for you. Please be sure to Contact Support if you have any questions or concerns. – Justin”

Hi guys, i´m dying here, i just logged into michaelkelly with some usernames and the info is there, i just need you to move from michaelkelly to my VPS ps86583, i know you already did it, but when i requested a move you were in trouble, and i´m seeing some restored info there, so, please, MOVE THEM AGAIN!!

@ DreamHost Team,

I hope to read someday that someone gets so fed up with your shit they go postal on you all.

Best of luck to you. I’ve already moved my site to another host.

GFY

I will still support DH because DH VPS package is still the most economy deal for me. So. DH, please still put and improve on your migration plan and services.

My site is still down. Almost 48 hours now with no updates from Dreamhost as to why they are still down. Actually working today on moving all my remaining sites off of Dreamhost.

@Daniel: I replied to you directly. I found your backups. Make sure you email me directly.

@Charlie: Thanks for the kind words. Changes are coming soon which will help make restores much faster.

@Braxton: I’m working on your VPS.

Could some one take a look at : ps54088 and try to bring it back to live, thanks.

ps58265 has been down for 3 days. Any word of when I can expect it to be back up?

When is ps47533 going to be taken care of???!?!?!? It’s on a different homie, but your status posting for that homie says the issues with it have been resolved so I assume you’re ignoring my comments there. My VPS (ps47533) was down earlier this week, then came back up with really old files, and now it is completely hosed. It doesn’t have a /home, and all sorts of stuff isn’t working. I’ve filed OMG priority support requests several hours ago and NO RESPONSE. I’ve tweeted your CEO, with NO RESPONSE. What is the deal?

Thats it, I’m done. Transferring my stuff to a new host. I’ve been a customer since May 2005, and the outage today cost me a customer. I’m shocked that its taken this many outages for them to drop me…

No more.
I’m done with the unanswered support tickets (No response for 9 hours, 5 minutes, 10 seconds today), all the while the DHStatus page is saying “All VPS guests are online” and “no reported issues”.
I’m done with the “fake root” access that I have to this “server”.
I’m done with the “oops, our bad” amateur mistakes being made constantly at DH (automatic updates, are you f%#!ing serious?).
I’m done with the constant failing raid arrays and excruciatingly slow restore process when something does break.
I’m done with the system-wide network outages, due to more of the “oops, our bad” mistakes.
I’m just done. Goodbye dreamhost. You’ve lost a long time customer due to this outage. Its been a long time coming. This was just the final straw…

So what goes around, comes around. My customer leaves me, so I’m leaving you. Here’s my 2 cents before I go… you should spend less time on stupid blogs and one-click installs, and more time making sure your servers remain online.

I realized something today. The ONLY thing that kept me at dreamhost for so long was the web panel. The web panel is great, when its up. But at the end of the day, it doesn’t make 2 $h!ts how good the web panel is if your server is down.

March 3rd, 2012 at 3:54 amdreamhost customer 2 Says:

Yep me too, all gone. I’m in discussion with MT who I think might match DH – there will be a lot of customers needing to swap I guess.

Hey INSANITY???,
http://www.linode.com
Just don’t host your bitcoin wallet on there. lol

Seriously though, I’ve got friends that talked me into linode. Their VPS’s have well over 100 days uptime on them, and one even has nearly 300 days. I’ve, personally, never seen my DH vps have more than 48 days. Thats the longest uptime I’ve ever seen it have. Typical uptime for my VPS is around 25-30 days. Oh, and its not like I’m in there rebooting it all the time, because I’m not. But someone is…

All of the servers I run for my business have many hundred days uptime as well. Not sure why DH can’t keep a server running for more than a couple months.

I got the Linode 1024 package. 1GB ram, 40GB storage, 400GB transfer for about $10 per month less than I was paying DH.

@David, Steban, Carlo: None of your VPS’s are hosted on this server but they are all fixed now and running without any issues.

@David: Justin personally replied to your tickets earlier today.

Ps6227

I lost the information that got into my server on 29 February and 1 March. To compensate for the fall of the servers for 50 hours straight, were operating as we begin to work long hours in the upload and images. All that work has been erased at once. And the load time of the web is so slow that it is impossible to navigate. An example from my server:
http://www.electropastor.com
Since the update server can not just browse, and hangs every few minutes.
In this website, we have deleted all the images that rose from the 29th to March 1st.

@david: I believe I restored some of the missing images, and replied to your ticket.

All fallen again. It’s a disaster. Last week, 55 hours does not work and some sites are not functioning normally more than 170 hours. Some of my websites with the new update going so slow you can not navigate and therefore are almost inoperative, information has also been lost. This is my undoing

Hi, I realize I’ve opened several complaints in recentdays. Had for years without claiming anything, but with the last update of the servers I’m having enough trouble on my websites. Of all the changes that were made the day of Feb. 29, just have recovered some photographs. The main problem with my sitesis extremely slow load time of each site, the 500 error is common and many can not browse web. I keep my serverupdates every day and it seems all servers are running smoothly. As an

> > example:
http://www.electropastor.com
http://www.electropastor.com/index.php?option=com_virtuemart&page=shop.browse&category_id=142&Treeid=12&Itemid=1

It has always worked well and since the server was upgradedstopped working, the load is so slow that you can not surf theweb. The error 500, the load is extramente slow and lost the data gained during the day Feb. 29. This client(www.electropastor.com) has given me a deadline today tosolve their problems or look for another client. It comprensibilebecause it takes more than 1 week with your Web inactive. A few hours ago I changed the database password to access the web and is inoperative.

Well this server should be a little quicker after today. I finished transferring my last 2 domains to linode, and canceled my DH account. Peace out!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver142 moving to new hardware and restoring from backups (RESOLVED)
10:28 am

Web VPS host homie-vserver142 moving to new hardware and restoring from backups (RESOLVED)

Posted (February 27th, 2012 at 10:28 am PST) by brians

Due to issues with the RAID controller on the web VPS host homie-vserver142, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support in your panel.

Please note, this is regarding the VPS host machine ‘homie-vserver142’ and no other services (email, MySQL) or machines are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

We will continue to update this post as information is made available. Please check back for updates.

Update Mar 14th, 2:50pm PDT: All VPS guests have been restored now! Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Oscar

Update Mar 14th, 4:53am PDT: All VPS guests have been restored. We’re still in the process of running one final sync, and double checking to make sure all files were restored. Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Chalise

Update Mar 13th, 9:21pm PDT: All VPS guests have been restored. We’re finishing up the data transfer, and double checking to make sure all files were restored. Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Dan J

Update Mar 13th, 5:30pm PDT: All VPS guests have been restored. We’re finishing up the data transfer, and double checking to make sure all files were restored. Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Dan J

Update Mar 13th, 11:15am PDT: All VPS guests have been restored. We’re finishing up the data transfer, and double checking to make sure all files were restored. Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Andrea

Update Mar 13th, 6:10am PDT: All VPS guests are reporting as online and it looks like we’re nearing the final stretch of the restore process. Thank you for your patience as this process completes! Please contact support with any questions or problems you may be having. – Chalise

Update Mar 12th, 10:20am PDT: The final VPS is still restoring, and all VPS guests are reporting as online. Thank you for your patience as this process completes! Please contact support with any questions or problems you may be having. – Justin

Update Mar 12th, 5:05am PDT: The final VPS is still restoring, and all VPS guests are reporting as online. Thank you for your patience as this process completes! Please contact support with any questions or problems you may be having. – Chalise

Update Mar 11th, 8:00pm PDT: There is only one more VPS guest to restore, thank you for your patience. All VPS guest machines are reporting as online. Please contact support with any questions or problems you may be having. -Andrea

Update Mar 11th, 1:45pm PDT: Restores of the last 6 VPS guests are continuing, now that the backup server is re-mounted. All VPS guest machines are reporting as online. Please contact support with any questions or problems you may be having. -Justin

Update Mar 11th, 10:00am PDT: The backup server is currently not wanting to mount over nfs. This means that the 6 remaining VPS to restore are currently not working or restoring. Our oncall admin is working on getting this resolved. All except those 6 should be working normally during this issue as they were restored to local disk already. Please contact support with any questions or problems you may be having. -Justin

Update Mar 11th, 6:40am PDT: The restore continues to run smoothly. All VPS guest machines are reporting as online. Please contact support with any questions or problems you may be having. -JJ G

Update Mar 11th, 3:50am PDT: The restore process continues to execute without any problems. VPS guest machines are reporting as online. Please contact support with any questions! -JJ G

Update Mar 10th, 5:10pm PST: This restore is moving forward and all VPS guests are reporting online at this time! Please contact support with any questions! -Oscar

Update Mar 10th, 5:45am PST: This restore is moving forward and all VPS guests are reporting online at this time! Please contact support with any questions! -PatrickM

Update Mar 10th, 1:13am PST: This restore is moving forward and all VPS guests are reporting online at this time! Please contact support with any questions! -PatrickM

Update Mar 9th, 7:50pm PST: This restore is moving along nicely and not encountering any trouble. All VPS guests are reporting online at this time, but please contact support if you have any questions or issues! -Dan J

Update Mar 9th, 2:30pm PST: This restore is moving along nicely and not encountering any trouble. All VPS guests are reporting online at this time, but please contact support if you have any questions or issues! – Oscar

Update Mar 9th, 4:11am PST: This restore is moving along nicely and not encountering any trouble. All VPS guests are reporting online at this time, but please contact support if you have any questions or issues! -Patrick

Update Mar 8th, 10:06pm PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Dan J

Update Mar 8th, 6:32pm PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Dan J

Update Mar 8th, 12:30pm PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Oscar

Update Mar 8th, 5:46am PST: This restore continues and all VPSs are reporting online! Please contact support if you encounter any issues, or have any questions! -Patrick

Update Mar 8th, 3:18am PST: This restore continues and all VPSs are reporting online! Please contact support if you encounter any issues, or have any questions! -Patrick

Update Mar 7th, 9:54pm PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Dan J

Update Mar 7th, 5:05pm PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Seohee

Update Mar 7th, 11:30am PST: All VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please contact support so we can help! – Oscar

Update Mar 7th, 4:50am PST: It looks like all VPS guests are currently up and running on this host and the restore on this server is moving along smoothly. If you do notice any issues with your VPS on this server, please  contact support so we can help! – Chalise

Update Mar 6th, 9:48pm PST: The restores on this server continue without any known issues. Please contact support if you notice any issues with your VPS on this host server! -Dan J

Update Mar 6th, 4:29pm PST: The restores on this server continue without any known issues. Please contact support if you notice any issues with your VPS on this host server! -dan w

Update Mar 6th, 1:39pm PST: All vps are up and working at this time, and the restore continues! Please contact support if you notice any issues with your VPS on this host server! -Brian S

Update Mar 6th, 10:30am PST: All vps are up and working at this time, and the restore continues! Please contact support if you notice any issues with your VPS on this host server! -Justin

Update Mar 6th, 3:52am PST: We are working on one VPS guest on this host, but all others are reporting online at this time! Please contact support if you notice any issues with your VPS on this host server! -Patrick

Update Mar 5th, 9:25pm PST: All VPS guests are up and running and the restore continues to execute without any problems. If you are running into any issues with your sites on this host server please contact support for help. – Dan J

Update Mar 5th 4:14pm PST: We are continuing to keep an eye on this restore. All VPS guests are up and running, but please contact support if you need assistance with your sites. – Andrea

Update Mar 4th, 9:29pm PST: We are continuing to keep an eye on this restore. All VPS guests are up and running, but please contact support if you need assistance with your sites. – Andrea

 

Update Mar 4th, 3:00pm PST: All VPS guests are up and running and the restore continues to execute without any problems. If you are running into any issues with your sites on this host server please contact support for help. – dan w

Update Mar 4th, 9:19am PST: All VPS guests are reporting as online and the restore continues to execute without issue. If you are running into any problems with your sites please contact support for help. – Justin

Update Mar 4th, 2:15am PST: All VPS guests are reporting as online and the restore continues to execute without issue. If you are running into any problems with your sites please contact support for help. — JJ G

Update Mar 3rd, 8:10pm PST: This restore is still running and all VPS guests are online. As usual, if you experience any issues with your sites or are missing data, please contact support for assistance! — Brian H

Update Mar 3rd, 1:50pm PST: The restore process continues and we are almost done restoring the guests that were missing data. Please let support know if you have missing data or if you run into any further issues. We will continue to post updates as the process continues. -Oscar

Update Mar 3rd, 5:35am PST: The restore process continues to execute and the host machine remains up and stable. Please let support know if you have missing data or if you run into any further issues. We will continue to post updates as the process continues. -JJ G

Update Mar 2nd, 10:05pm PST: It’s come to our attention that certain VPS guest machines are missing data from the restore that was completed recently on “homie-vserver142″. For that reason we are re-opening this status post. Our system administration team has already mounted backups via NFS for the VPS’s missing data. We have already started to restore all data from the backup server to the new server as well. At this time all VPS guests machines should be working. If you notice your VPS is not working or missing data, please contact our support team and we can look into it for you. -Dan J

Update Mar 2nd, 3:15pm PST: It’s come to our attention that certain VPS guest machines are missing data from the restore that was completed recently on “homie-vserver142″. For that reason we are re-opening this status post. Our system administration team has already mounted backups via NFS for the VPS’s missing data. We have already started to restore all data from the backup server to the new server as well. At this time all VPS guests machines should be working without any issues. If you notice your VPS is not working or missing data, please contact our support team and we can look into it for you. -Oscar

Update Mar 2nd, 12:36am PST: This restore is now complete! All services are up and running, but if you are experiencing any issues with a guest VPS on this host, please do not hesitate to contact support for assistance! Thanks for your patience during this restore! -Patrick

Update Mar 1st, 5:40pm PST: The restore continues on this server. If you have any issues with your VPS on this host machine, please be sure tocontact support so we can take a look. -dan w

Update Mar 1st, 10:00am PST: We’re having a few issues with 2 VPS guests on this host machine. Otherwise, the restore process continues and all other VPS guests are up and running. If you have any issues with your VPS on this host machine, please be sure to contact support so we can help. – Justin

Update Mar 1st, 3:50am PST: We’re having a few issues with 2 VPS guests on this host machine. Otherwise, the restore process continues and all other VPS guests are up and running. If you have any issues with your VPS on this host machine, please be sure to contact support so we can help. – Chalise

Update Feb 29th, 5:55pm PST: The restore continues, we have addressed the slowness on the backup server, the restore process is now moving much faster. If you have any issues with your VPS on this host machine, please be sure to contact support so we can take a look. – Dan J

Update Feb 29th, 9:20am PST: The restore continues, but the backup server appears to be responding slowly and this is causing some slowness. We have an admin looking into the issue. If you have any issues with your VPS on this host machine, please be sure to contact support so we can take a look. – Justin

Update Feb 29th, 4:22am PST: All VPS guests on this server are up and running and the restore continues to run smoothly as well. If you have any issues with your VPS on this host machine, please be sure to contact support so we can take a look. – Chalise

Update Feb 28th, 8:55pm PST: The restore was restarted and running well, and all but 3 VPS guests are reporting up at this time. Our admins are working to get the VPS guests and restore back up as fast as possible. Thank you for your patience. We will update with more information as we have it. – Seohee

Update Feb 28th, 6:48pm PST: The restore is currently halted due to an issue with the backup brick, and all but 5 VPS guests are reporting up at this time. Our admins are working to get the VPS guests and restore back up as fast as possible. Thank you for your patience. We will update with more information as we have it. – Seohee

Update Feb 28th, 9:30am PST: The restore continues to run, and all VPS are reporting up at this time. Our admins are continuing to work around the clock to get this restore finished up and all services back to their normal speed. Thank you for your patience and we will update with more information as we have it. – Justin

Update Feb 28th, 5:10am PST: The restore continues to run, but our admins are still hard at work bringing some of the remaining VPS guests back online. Our admins are continuing to work around the clock as fast as possible, though we sincerely apologize for the delay. Thank you for your patience and we will update with more information as we have it. – Chalise

Update Feb 27th, 8:46pm PST: The restore to the new hardware is running, but there are still some VPS guests that need to be manually brought back online. We’re working as fast as we can, please accept our apology for the delay. We’ll update this post with more information soon. – Seohee

Update Feb 27th, 4:53pm PST: The restore to the new hardware is running, but there are still some VPS guests that need to be manually brought back online. We’re working as fast as we can, please accept our apology for the delay. We’ll update this post with more information soon. – Andrea

Update Feb 27th, 12:25pm PST: The new hardware is online, and we’re starting to mount the backups, so it can serve your sites while the file system check is still running on the old hardware. Your site will be online shortly. We will continue to post updates as information is made available. Thanks for your continued patience. -Brian S

Update Feb 27th, 11:11am PST: We’re getting close to being able to serve your sites again, from backups. Sorry about the delay. We will continue to update this post as information is made available. -Brian S

This entry was posted on Monday, February 27th, 2012 at 10:28 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

69 Responses to “Web VPS host homie-vserver142 moving to new hardware and restoring from backups (RESOLVED)”

ps20199 ps22135 ps22540 ps37457 ps38312 ps38380 ps38397 ps38409 ps38413 ps38498 ps38606 ps38960 ps39540 ps40088 ps40095 ps41519 ps41865 ps41922 ps41966 ps43484 ps43766 ps44595 ps44599 ps44700 ps46853 ps47533 ps47587 ps49144 ps50545 ps52822 ps54088 ps54092 ps54095 ps54097 ps54457 ps54590 ps55323 ps55353 ps55383 ps55801 ps56607 ps56681 ps56768 ps57965 ps57986 ps62621 ps65065 ps70325 ps71986 ps72190 ps73045 ps79884 ps80523 ps85656 ps86624 ps86635

ps63287 is not included above yet my site has been down now for 3 hours.I have sent three tickets with no response and no action.

still down, no one from support is getting back to people.

@Steve – Please include some details, so we can look into your case.
@isaac – your VPS guest is not listed because it’s not on this host, and your host isn’t reporting any issues. I’ll look into why your guest is down and get back to you soon.

my site has been down all day as well (lovitt.com) and e mailed and tweeted and support several times and nothing in response

ps25602 down again.

ps47728 is down.

it’d be awesome if you guys responded or something.

My VPS, ps47533, is on the list. I notified you of its outage at 8:17 AM PST this morning, after spending a half hour on my own trying to reboot the server. You did not make any reply. Now, four hours later I find this, and still no contact from Support. As Kieth said, it’d be awesome if y’all responded to support requests. I filed this with a top priority indication, and if four hours of no response is your idea of responding to top priority support requests then maybe I should be looking elsewhere. What do you think?

another hour gone, more dollars out the door for my business

I’m in the exact same boat as David Herron. My server, ps30407 doesn’t appear on this list, but my sites are down as well. I opened ticket 4776538 at 0648 PST, and have had no response. Pretty unhappy at this point.

Our server is also down and not in your list.. ps63949. I tried restarting the server and that didn’t help.

ps75586, I have about 25-30 sites fully hosted and all of them have been down for 24+ hours.

@Brian I had to start the apache server. Your site is back online

@Nathan @Keith Your guest is not hosted on this the vserver host. Please submit a support request with the details of your issue

@David Post has been updated with the latest details. We will continue to keep everyone in the loop.

@Sheryl I will address and respond to your support request

My sites are on ps27304 and have been down since yesterday. Not a word from support, and it’s not on this list. Please help!!!

ps39386 down 24+ hours

What about our site? We are on ps81981
no answers to any support tix. day 3 of outage!

Hi-
My site has been down since Sat. You do not list it. So frustrating. Lots of business lost for my clients.

ps30401 is also down and not listed. please fix asap

My site has been down for 2 days now… it isn’t listed as well… ps27113

@Alex and Marcelo – your machine is up, and so are the couple of sites I just tested. If you need further assistance, please contact support through the DreamHost panel with the specifics.
@ogp – we’re on the phone with you now, your case is memory-related
@Andrea – your machine is listed under http://www.dreamhoststatus.com/2012/02/25/web-vps-host-server-%e2%80%98homie-vserver173%e2%80%b2-moving-to-new-hardware-and-restoring-from-backups/ and is likely be manually restored. Sorry about the delay.
@Michael – Your VPS is up, and I restsarted the http service, just in case. I tried to check some sites to verify, but you have a lot that aren’t registered. If you need further assistance, please contact support through the DreamHost panel with the specifics.
@Tony – you are on another, unaffected server, but your apache service wasn’t running. I fixed it for you, your site is loading now.
@David Herron – your VPS guest is among those that still need to be brought online to start the restore. We’re working on this as fast as we can, sorry about the delay.

when are you expecting the manual restore on ps41422 to be complete?

Still down on 72190

how long is this goig to take?

41966 is still down!!

this is rediculous!!

This is ridiculous. Why do we pay extra for worse service? What is taking so long. This is a common problem with your systems, I figured you might actually have put in place something to at least speed up recovery, let alone actually stop the problem before it happens.

Dreamhost you are slipping rapidly from being good hosting down to horrible hosting.

Server: ps71986

Hi,

I can see my server on the list, my site was down nearly all day yesterday and it’s still down. Will this be resolved today? I tried contacting support but I still haven’t received a response yet.

ditto ditto ditto .. this is <bleeping/> ridiculous

stupid, horrible, ridiculous, awful, anyone got a thesaurus?

ps50502 has been down for 3 days now. I’ve restarted the server many times hoping it would come back up. I’ve tweeted and submitted a ticket with no response back. It would be great to get an update.

The backup for http://www.expenews.com you charged in ps25733 is a old version of our applications from TWO MONTS AGO!!!

When is ps54590 being brought back up?

came back with an outdated backup this morning- site’s gone down again- no word from anyone- this is bollocks

@ALL Our admins are diligently working towards bringing your VPS guests online. We sincerely apologize for any inconvenience this has caused. Your continued patience is greatly appreciated.

Ho boy, this looks messy. Here’s a question, the other day on a regular webhost about 4 hours of work I did setting up someone with WordPress custom configured disappeared as if it was wiped out. Not my VPS service, but it was majorly inconvenient as we didn’t have a full backup for the site, literally it had been running for less than a day before it got wiped.

So, now that my VPS is out of commission here and there are people coming back up with old backups of data, what exactly is happening with our VPS’ are they going to come back as they were or shambling reanimated corpses that sort of look like what they were?

For the record – we’re well into the 2nd day of outage for my site. Do you guys have any idea what this is costing? And it’s not just this thread of griping about this outage, but look through the other outage reports and it’s easy to see plenty of other gripage. Dreamhost is losing it.

i think some kind of compensation is in order

Ok, now you have cost me a day of my international holiday, business and credibility. Thank you for screwing everything up. I am up at 4am trying to solve this mess to keep my clients happy.

Server: ps71986

@Chalise,

My server is still down, I already contacted you but had no response I just sent another support ticket since you advise us to contact you on your latest update but I still haven’t heard back.

Thankfully I did not wake up this morning to a 3rd day of downtime on my VPS (ps47533). HOWEVER, .. all is not right. I have filed support requests (as you requested above) for:- a) the website is responding EXTREMELY SLOWLY… b) recent changes as old as 3 weeks are not present

@Chalise, @Justin, @Anyone! .. can you please check out ps37457?

I have been following that thread for 3 days now and every time someone posts that it is fixed it doesn’t work. I understand you guys are seriously swamped, can you at least reply to one of my support tickets or posts? I need to know when we can access data at the very least. The blog posts say to contact support, which we have done. It looks like you are ignoring people and only responding to others. I am the primary account holder on the VPS. Please advise!

@Justin

I’ve been trying to contact support for 2 days now and I still haven’t got a response. This is the only place were I can get my message across.

My server is still down, can you please take a look at it – Server: ps71986

You guys have an interesting interpretation of what it means for a VPS to be up. ps54590 has been down this entire time, and inaccessible. None of my accounts work, apache says it’s not configured with my domains when I send a support request. Please provide a status update immediately. I want to know what’s going on. This delay in communication is raising my ire.

@ John: All sites should come up with the data that we had in our backups prior to the server crashing. Contact support if you need specific information.

@ abu: There was a misconfiguration on your servers apache. It’s fixed now and your site is up and running.

@ weknowsnow: Tested several domains and they all seem to work. Your support ticket has already been responded to as well.

Server: ps71986

@dan

Thank you for getting my server back up, however I can still see the misconfiguration errors on my admin section, when I try to access mywebsite.com/wp-admin/ I get the following message:

“Warning: Cannot modify header information…”

Can you please take a look.

@dan

Thanks for getting back to us on this, though my server is still down. Is there an ETA, or a status as to what is going on?

Dan, are you telling me there were no files in the backup??? that all my stuff is gone then???
are you serious???

#4783395 are you guys currently restoring my ftp?

Can you guys tell us what is going on please?

Server: ps71986

My website is up and running but it’s not back to normal, the misconfiguration on my servers Apache hasn’t been fixed 100%. I can’t access my WordPress admin, I keep getting error messages this problem wasn’t there before can you guys please resolve this ASAP. I’m running out of time, I can’t afford to keep on waiting.

This ridiculous process is still bonkers… my VPS (ps47533) is still slow, and worse hasn’t been updated with a proper restore still. The files on the server are out of date by 3 weeks or so. I’ve filed service desk requests on this, but only gotten a response on the initial one I filed on Monday. Is this your idea of service?

Hi, so I see my files are partially restored, not completely, but only partially. What I see is a bunch of the old folders where my files used to be are owned by root. Are these directories/files gone? Or are they going to be completely restored? Thanks by the way for bringing the server back up, however I have not received a single response to my support tickets, so I haven’t tried to address this particular issue in a ticket either.

Please advise!

@John P: Checking out your VPS.

server is out again geting a 403 error Forbidden
You don’t have permission to access / on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
can’t FTP in either. about to walk into a client meeting. Thanks Alot!

I was able to fix that error myself in a bunch of directories that were owned by root:root. After I changed their permissions to my user (like the other files and directories that worked) I was able to use most of my restore. There are some things missing, but not all.

@Justin Thanks for checking it, do you mind reporting something to my ticket?

– J

This is complete and utter bullocks … the restore is not complete, and this is not resolved. My VPS ps47533 is a complete and utter mess right now – I’ve sent in a support ticket, not received any response despite it being hours ago, and my site is still down.

@David Herron: your server should be working again now

@Dan J – no, my server is most definitely not in a properly functioning state. First, the files are extremely stale. Second, the site at visforvoltage.org is misbehaving badly. I have responded to the support desk email with other information.

DO NOT CONSIDER THAT MY SITE IS FINISHED BECAUSE IT IS STILL BORKED AS FAR AS I AM CONCERNED!!!!!!!

As for your status log – let it be known for the record that your report this morning at 5AM saying that all was fine – that’s a total lie. My server was completely borked this morning, it had an empty /home directory and was missing other things. How in the world you could have thought all was fine is beyond me.

@David Herron: I apologize that we didn’t notice that your site was missing files. The restore appeared to be complete this morning when we reported the all clear, but obviously it turned out that the restore had missed many files. That is why we reopened this post and have been restoring your files. Your VPS is still mounting the backup server which is why you’re seeing stale files.

Okay guys, so you have us on a backup server and my home directory is somewhere else, I can’t FTP or SFTP or shell into my own account to work on things. I have some major devel work to do as of … last week and from the sounds of the last support mail I finally got an answer for, you’re going to move us back from the backup server to even more hardware. Is that transition going to be as messy as this one? Please start fessing up to what’s going on.

@John P: I sent you a reply through email with the details, but your user config is corrected and I’ve tested sftp/shell access. Let us know if you keep running into trouble. As for the move to new hardware, we are actually continuing the restore to the same machine we had restored to already. Some VPS machines had been missed and those need to be restored; we are not changing the hardware again.

@jj @DanJ

Can you please check out ps37457? My VPS has been down for almost a week now. I can back up for a day with the wrong files, I have sent a couple of support tickets with no reply. Can you please check out this VPS and help me get it back online? PLEASE!

@weknowsnow: Your VPS is up and running and the restore has been completed on your VPS. Can you please check your sites/server and let me know if you are having any issues with your sites being down?

So, the host seems to be about 10x slower than previous to this drama, how is that being addressed?

@John P – you’ll likely notice intermittent slowness until the restore completes, sorry about that.

My server is effectively down again .. the VPS is actually running, but no services are running. I have no idea how you could say in the status that “Update Mar 11th, 6:40am PST: The restore continues to run smoothly. All VPS guest machines are reporting as online. Please contact support with any questions or problems you may be having. -JJ G”

Reading the resources graph on my VPS it looks like the system stopped working at Midnight last night. It is now 9:24 AM .. so, that’s 9+ hours of downtime, and you idiots report that “all guests are reporting as online”. HUH?!?!?!? How the flark can you say that? My VPS is effectively down, has been for 9+ hours, and you say all is fine?

What kind of idiots do you think you are?

And, I have filed a support request, in the OMG level of priority. I wonder how many days it will take you to wake up from your nap and take care of this?

@David: Your VPS was not reporting on our monitoring system, which checks all VPS for ping, apache running, and a few other services. Individual site problems may not be detected, if apache is responding normally. The restore had also not reported an issue until recently after the last status post. I verified that there is a nfs connection issue, our oncall admin is restarting the server and the restore now, and your VPS should be up shortly. I’ll reply to you momentarily when your service is restored.

My VPS is also gone. I am unable to get any work done and I do have some development work to do, besides the fact that all my sites are down. I am so glad I’m not hosting any paying clients here at the moment because this is just unacceptable.

@ John P: Looks like there was a misconfiguration issue with your users that was preventing apache from starting. It’s all fixed up now.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)
8:35 pm

Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 25th, 2012 at 8:35 pm PST) by Brian@dreamhost

Due to issues with the drives on the web VPS host server ‘homie-vserver234′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. — Brian

Update Mar 3rd, 8:50pm PST:  Ok, we’ve received word that this server should now be fully restored. We’re marking this issue as resolved, so if you’re still seeing missing files, please let support know ASAP, and we can look into the issue for you! — Brian H

Update Mar 3rd, 8:10pm PST: This restore is still running without any issues and all VPS guests are online. The restore is getting close to finished, so we’ll keep you updated. If you experience any issues with your sites or are missing data, please contact support for assistance! — Brian H

Update Mar 3rd, 1:55pm PST: All VPS guests are up and reporting as online. The restore is running without any issues. Please let support know if you have any issues or questions. -Oscar

Update Mar 3rd, 3:45am PST: This machine remains stable and all VPS guests continue to report as online. Please let support know if you have any issues or questions. -JJ G

Update Mar 2nd, 10:05pm PST: All VPS guests on this host are now up and running! The restore has been running smoothly since it was restarted this afternoon.  Please contact support if you experience any issues with your VPS guest on this host server! -Dan J

Update Mar 2nd, 3:15pm PST: All VPS guests on this host are now up and running! The restore process was hung and our admin team had to temporarily stop it. It has now been restarted and is running smoothly without any issues.  Please contact support if you experience any issues with your VPS guest on this host server! -Oscar

Update Mar 2nd, 5:44am PST: All VPS guests on this host are now up and running! The restore is moving along and we are showing no other reported issues. Please contact support if you experience any issues with your VPS guest on this host server! -Patrick

Update Mar 2nd, 1:25am PST: All but 1 VPS guest on this host are online at this time! We are working to resolve the issue with the 1 guest, but the restore is moving along without any other issues. Please contact support with any questions! -Patrick

Update Mar 1st, 5:40pm PST: All VPS guests on this machine are up and running and the restore is running once again. If you notice any issues with your VPS, please don’t hesitate to contact support. More updates to come as the restore progresses. – dan w

Update Mar 1st, 10:00am PST: All VPS guests on this machine are up and the restore is running once again. If you notice any issues with your VPS, please don’t hesitate to contact support. More updates to come as the restore progresses. – Justin

Update Mar 1st, 4:51am PST: We resolved the access issues with this server and the restore is continuing without issue! All VPS guests are online at this time, but please contact support with any questions or issues! -Patrick

Update Mar 1st, 4:38am PST: The restore is continuing, but the server has become inaccessible over SSH. Our Admins are looking in to this issue at this time and we hope to have it resolved shortly! Please contact support with any questions! -Patrick

Update Feb 29th, 6:34pm PST: The restore is going well, and all VPS guests are running. If you notice any issues with your VPS, please don’t hesitate tocontact support. More updates to come as the restore progresses. – Seohee

Update Feb 29th, 9:30am PST: The restore is going well, with the server load low and all VPS running. If you notice any issues with your VPS, please don’t hesitate to contact support. More updates to come as the restore progresses. – Justin

Update Feb 29th, 4:30am PST: All VPS guests on this machine are up and the restore is running once again. If you notice any issues with your VPS, please don’t hesitate to contact support. More updates to come as the restore progresses. – Chalise

Update Feb 28th, 6:40pm PST: All but 3 VPS guests are running, and the restore is currently at a halt due to some issues with it connecting to the backup brick. We are looking into this and hope to get the VPSes and restore back up as soon as possible. We appreciate your patience. We will continue to monitor the situation and update this post regularly as we have more info. If you notice any issues with your VPS, please don’t hesitate to contact support. – Seohee

Update Feb 28th, 9:30am PST: All VPS guests are running and the restore is continuing. We will continue to monitor the situation and update this post regularly as we have more info. If you notice any issues with your VPS, please don’t hesitate to contact support. – Justin

Update Feb 28th, 5:05am PST: We’ve noticed a few VPS guests that are having a few issues; our admin team is working on bringing them up. Most guests are up and running, however, and the restore is moving along smoothly. If you notice any issues with your VPS, please don’t hesitate to contact support. More updates to come! – Chalise

Update Feb 27th, 5:55pm PST: The restore on this server is still moving along smoothly. All VPS’s are running to our knowledge. If you notice any issues with your VPS on this machine, please don’t hesitate to contact support. We’ll update this post again soon as the restore progresses. – dan w

Update Feb 27th, 12:22pm PST: The restore is moving smoothly, and all but one VPS guests on this host should be up and running. The one still affected VSP ( ps37800 ) is getting looked into. We’ll update this post later today with more information. If you have any questions or concerns, please don’t hesitate to contact support – Brian S

Update Feb 27th, 8:00am PST: It looks like one guest on this host machine is having problems coming back up, which our admin is working on fixing. Other than that, all VPS guests on this server should be up and running. More updates to come. – Chalise

Update Feb 26th, 11:50pm PST: All VPS guests on this server should definitely be up and running, and the restore continues to run smoothly. Just a reminder – affected VPS guests can be found in the first comment below this post. Thanks again for your continued patience! – Chalise

Update Feb 26th, 5:25pm PST: All VPS guests should be up and running at this point. If you’re still having issues with your machine, or if the machine is accessible but your sites are loading with the wrong content or not at all, please try a restart from the DreamHost panel and contact support if you need further assistance. We appreciate your patience while we’re completing the restore. – Andrea

Update Feb 26th, 9:30am PST: VPS services are being brought back up, we still have several VPS guests down and are working as quickly as we can to resolve this issue. Further updates will be posted as we progress through this situation. – JJ G

Update Feb 26th, 6:10am PST: Networking has been restored to the host machine. Our administrators are still working on bringing up the VPS services. Thank you for your patience and we’ll continue to post further updates. – JJ G

Update Feb 26th, 5:25am PST: We are currently experiencing networking problems on this host machine and guest services are not responding. We have the administration team on the issue and will post updates as the situation changes. – JJ G

This entry was posted on Saturday, February 25th, 2012 at 8:35 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

95 Responses to “Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)”

List of servers affected:

ps18424 ps18686 ps21403 ps22719 ps36138 ps36837 ps37718 ps37800 ps49609 ps49641 ps49659 ps49692 ps49791 ps49839 ps49847 ps49874 ps49922 ps50141 ps50178 ps50179 ps50652 ps50653 ps50654 ps50655 ps50656

What about ps81981 — we have been down almost all day!

Is it just me or are there others who are not able to access their Dreamhost panel? In fact, even pinging dreamhost.com is timing out.

My site has been reverted back to last week – all updates gone. One click install has gone, all blog posts gone. What the hell are you doing there has it become pot luck who does what. Do you get bored and start randomly pressing buttons? You’ve removed copyrighted material.

@Craig: Your sites are running off mounted backups. We should be able to restore your current data from the old hardware once it has finished running a file system check. This will take hours to complete, but we should be able to restore it for you. I sincerely apologize for the inconvenience that this causes. Submit a ticket and we will restore your data as soon as we can.

Oscar, please reply to me. The site (ps81981) has been down all day, what’s happening?

7h up in the last 50hours…… THIS IS NO POSSIBLE!!!!!!!

Posting in the comments here IS NOT an official way to contact DreamHost.

7h up in the last 48hours…… THIS IS NO POSSIBLE!!!!!!! My Ps is ps21925… I sincerely apologize for the inconvenience that this causes…

Hi,

I’m on stampeders and my site is completely down, just a blank page, I’ve sent through a support ticket, anyone else having the same problem?

What a nightmare Dreamhost has become!!!

PQktOn msbsggoqjtdz, [url=http://zvoliaqqhkxr.com/]zvoliaqqhkxr[/url], [link=http://owgcalmljxfl.com/]owgcalmljxfl[/link], http://fllttjrsizsh.com/

yes SRSLY…

of course I also opened ticked,
replied on twitter.

…Should I travel to USA?

ps46383 is down as well.

My server is not listed on any of the hommie servers listed (234, 173 or 127) and all of my DHost websites are down. Please be aware and fix this ASAP.

My server is not listed on any of the hommie servers listed (234, 173 or 127) and all of my DHost websites are down. Please be aware and fix this ASAP.

SERVER ps29410

My VPS MySQL server is down, none of the 6 sites I host are working without mysql. I submitted a ticket.. psmysql47950

Our server not listed but also has been down for over a day: ps80498.

ps85428 not responding

My site has been down as well and I don’t have VPS as far as I know. How can I know if my problems will go away when the rest of these do?

about 4day you should wait untill working again.
most server never working and dead.

@Ernesto: It’s true your server isn’t on any of the actively restoring machines. In fact it seems to be up and running from the time you last commented. If you are having issues with it please submit a ticket to support:

https://panel.dreamhost.com/index.cgi?tree=support.msg&

@derp: It looks like your mysql service is responding as it should, uptime on your service looks good too. You may want to contact support as well.

https://panel.dreamhost.com/index.cgi?tree=support.msg&

Ok WHAT about server PS81981??? This is the third time i am asking. Also, I opened a tix ages ago and have hear no word. Please check the site

I have 10 sites down for more than 24 hours. That is pretty unacceptable. Deplorable even. This cost your customers a LOT of money, and far worse, credibility and customer loyalty. And Is it even possible these precious resources will be LOST????

I’ve been on Dreamhost almost 5 years. I’m afraid they’ve lost their way. We’ve seen regular and long stretches of downtime over the last 6 months. It’s clear Dreamhost is not a sufficiently professional operation for commercial sites to rely on.

We’ve been on a Dreamhost VPS, paying around $60 a month. We’re moving to Amazon EC2, which has proven far more stable and is a better deal in that range.

Sorry Dreamhost employees – please get on your bosses to get serious about the criticality of service they provide — maybe the company can recover its reputation in a few years.

Thanks for the reply Oscar. I have submitted four tickets, one for each of your mistakes. I have had no reply after 24 hours now. If my sites aren’t fully restored to the state I left them in I will be expecting significant compensation.

This is kind of interesting. I see a whole lot of vservers having RAID or “drives” issues.

Does this mean:

(a) DH got a bad batch of servers

or

(b) All of our vservers are running on the same vast machine, and it’s having drive troubles

We are also down http://www.feneal.com and have been for over 24 hours. No response from support ticket I placed early this a.m. ? Is there anyone at dreamhost assigned to customer support. Live chat has been dead so long, you should take it off your support pg. Not good, people.

Why oh why can’t you guys send those of us who are on the affected servers a notice that our sites are down and why they are down. That would really help to calm folks down. Really, this is a no-brainer…you know who occupies those servers, so just tell us!

Seriously, can we get an answer to that?

Thanks,
Matt

P.S. my server is: ps34651

ps5796. Ditto all of the grumpy comments above – have been with DH for a long time now but this is starting to get ridiculous. What is the point in Live Chat if you don’t staff it? How can this have happened again such a short time after the last screw-up, and in light of the “We’re really sorry, this won’t happen again” newsletter? I don’t think we can leave any paying customers on servers that have this kind of downtime.

Maybe they take Sundays off…

Still no response to my ticket from four hours ago…Dang.

We got an email saying our site was up and the VPS has been restarted but it’s still not up! Is this a joke??

All VPS guests should be up and running at this point. If you’re still having issues with your machine, or if the machine is accessible but your sites are loading with the wrong content or not at all, please try a restart from the DreamHost panel and contact support if you need further assistance.

ps80498 – this is absolutely ridiculous. Site now down for two days.

All my websites on ps27142 are still down. I tried to reboot them but still nothing. Please help!

Andrea, How does one re-start from the DH panel??
I also contacted support (after they informed us our site is up but it wasn’t) and haven’t heard back for 5 hours! Seriously.

We haven’t heard back from support for 1 day and 8 hours now. VPS is still down (ps40898). Tried re-allocating more RAM, tried rebooting the server. Panel claims it succeeded in rebooting the VPS, even though I see no evidence of that. Can’t use SSH. Getting frustrated.

Rebooted server. Still crapping out. No answer from support. The other web hosting companies I’ve contacted got back to my right quick…

Now there seems to be some trouble with mysql too

I have been with DH for more than 4 years, 2012 has been terrible (no wonder they say doomsday is around) , Since my site is down and nobody from support is replying (or willing to fix it) I have got plenty of time which I’m spending on researching and comparing good webhosts

ps25733 also down, 40% uptime in the last three days!!! I did the suggested reboot from control panel but that didn´t solve the problem. Tickets unanswered for more than one day.
PLEASE SOLVE IT!

@Dana, @Matt, @Mike, @OGP, @CBC, @SL, @Camilo – your servers/sites you’ve referenced are on this host machine and are not related to any issues regarding homie-vserver234. The first comment includes the list of affected servers. If your VPS is not listed there, your issue is not related to this post. If you have already submitted a ticket, we have received it and will get to you as soon as possible.

If you have not yet submitted a ticket, you can log into your panel and go to Support / Contact Support to submit a support ticket, here:
https://panel.dreamhost.com/index.cgi?tree=support.msg&

Lots of us having the same problem with VPS ……… for me, the site is sometimes responsive, but slow, intermittent. It is an OpenX install. I can navigate my way around the admin panel, but it is not delivering ads, or else very slowly after a long hangup it delivers the ads .. but meanwhile the rest of the websites don’t load because they wait for the ads. So until this is fixed my websites have no ads because I disabled them from showing. FTP connects after a few tries, but then disconnects when trying to change directory or open/edit a file. DreamHost says they can connect to the site just fine and can’t duplicate my error, but they aren’t looking at it properly. It isn’t just an issue of connecting to the server one time, it is a matter of processing dynamic or extended connections, maintaining uninterrupted stability long enough to figure a php or deliver an image file .. that is the problem. My entire day has been spent just getting DH to consider the issue like that rather than simply trying to make a single connection to the domain and then hours later again tell me that they can’t understand my issue. Duuuuuuhhhhh what a waste of a day! Come on man, you think I’m just sending a support ticket cause it’s all good? Sorry DH I love you but it’s true. Sunday lala land or something over at headquarters

Also to clarify when I said “I can navigate my way around the admin panel” I ought to have said that it is very very slooooooooow, the browser status just keeps spinning spinning for beyond 10 seconds, sometimes 30 seconds, then finally spits out the page. This never happened until this morning. It’s problematic, something isn’t right. Will we have the courage to face that truth? @DreamHostCare, are you heroic? Or technological cowards?

I know this isn’t the support ticket area of DH, believe me I already have a ticket going for this. And I know my ps isn’t in the list for this blog post, but there isn’t a blog post for my ps and anyway obviously there are other pses having issues so until this blogger comes to terms with that fact and makes a more general post covering more pses or just the state of things in general, this is probably the best place to contribute my experience on the ps problem for the greater good of the community. Because who will see my support ticket besides the tech guy in denial that there’s a problem? If he can’t or won’t look seriously enough to fix it, what do I do? I will have to reach out to these folks visiting here and reading this who are most likely suffering along with me for some spiritual support and suggestions for more reliable web hosting (although lord knows I’ve tried so many and there is no such thing as a reliable web host). Someday when I am twice my age now I will host my own websites on my own machines, straight to the grid!

of course chat is offline right when everyone’s sites go down. tech support has been very slow to respond since we went down nearly 2 days ago.

@Chalise My server PS59768 is down too, not listed here though

shit man…looks like the world is ending…at least that is what it feels like, because I CANNOT CONTACT DH!! Oh yes, two of my PS’s are stuck some where in space I think?

I meant to also say/comment on Joe’s comment…I am with you in spirit okay!

All my websites have been down for the last two days!! PLEASE SOLVE IT ASAP!!

Chalise,

Your response is unacceptable. I have tried to contact you every way possible and my site, upon which hundreds of teachers, students, and parent depend is still down and I have not heard from you for 24 hours.

Get the job done.

All my websites on ps27142 are still down, after two days. Submitted a ticket yesterday, still have heard absolutely nothing from Dreamhost.

10 hours waiting still there is no response for my ticket. oh DH what happened to your service..

my client are so angry with me it kills my business..

ps49922 working but 2 of directory of my site was changed the permission to ‘root’ which caused error to my site.

Hey Dreamhost–all of my sites on ps22161 are still down.

I get this error for each user on my account:

“Your home directory does not exist on your web server.”

I get this error for each website on my account:

“Your website does not appear in the Apache configuration file.”

Any idea when this will be fixed?

JUST WAITE UTILL ALL DOEN

I’m now at 13 hours without a response to my ticket. Also on VPS, have no idea what my server number is. At the very least a response saying “your server is affected and we’re looking into it” would seem appropriate.

This is killing me. Not getting any 503 redirects anymore. Just crapping out. Been almost 24 hours. Please stop staying “everything is good” and tell us what’s happening.

dreamhost “where nothing ever work”

Hi, I’m looking for help on ticket #4775731. All my websites have been down over 24 hours. I am VPS ps27142. I have entered two support tickets, contacted you on twitter, tried chat support (it’s not live) and still heard absolutely nothing. Please help!

What is going on? Even my comments on this page are not working.

Site is down for more than 24 hours with no email from support… sad.

Our site as well as 45 of our client’s websites are down. We first reported this problem 4 hours ago. There is no way of getting hold of you people and no updates on the issue. Terrible customer service. You guys are no GoDaddy!

NightmareHost.com is unavailable but I would suggest you guys keep tracking this domain as it makes a lot of sense for your business

Repost from a thread seen above…EVERYONE should read this and move their host to another company

I’ve been on Dreamhost almost 5 years. I’m afraid they’ve lost their way. We’ve seen regular and long stretches of downtime over the last 6 months. It’s clear Dreamhost is not a sufficiently professional operation for commercial sites to rely on.
We’ve been on a Dreamhost VPS, paying around $60 a month. We’re moving to Amazon EC2, which has proven far more stable and is a better deal in that range.
Sorry Dreamhost employees – please get on your bosses to get serious about the criticality of service they provide — maybe the company can recover its reputation in a few years.

BOYCOTT DREAMHOST!!!

This is absolutely disgraceful. All three of our commercial websites have now been down for approx 24 hours costing us a great deal of lost revenue. This situation is bad enough, but Dream Hosts attitude towards our problems have compounded the matter. We have been in contact with you all day only to be continually told that “everything is now up and running”, well it isnt?, do you think if you say it enough times the sites will re-appear?

What kind of Micky Mouse operation are you people running?

no one will be fired over there as no one ever is and nothing ever changes. please go out of business as at this point wouldn’t even notice the difference

@Joe Our Tech was not able to replicate the behavior that you are reporting. If you are still having problems, please reply to our support response

@Malcolm @Ady @Camilo Could you please provide us with your VPS ( ex. psXXXXX )?

@Loki @Matt Your VPS guest is back online. Please let us know if you are still having problems

@Mike @Steve Your VPS guest is online and sites are successfully loading

@John1 The permissions on your directory have been corrected

@Jeff The webserver needed a restart. Your site are back online.

and what about us?

My site has been down for most of the last two days and came back up briefly today, but it wasn’t displaying correctly and was really slow to load (my VPS is ps37800)

Each email I get from you is just telling me the site is back up online, when it’s not. I’ve rebooted the server multiple times (11 times in the last two days) as well as increased the RAM to as high as it will go, and still nothing, and my FTP is still inaccessible.

How much longer until this is fixed? It’s getting ridiculous now.

We are PS39975!!

why do you guys keep posting “please don’t hesitate to contact support”? i think it’s more appropriate for us customers to say “please don’t hesitate to respond to customers support tickets”.

no response in 24 hrs, and barely any communication for 3 days.

When you say that all VPS’s are up and running on this machine, does that include mine (ps37800)? Because if so it’s certainly not up or running, and we’re almost at two and half days of downtime now.

There’s FTP but no web access.

ps39414 is down as well!

Still down, closing in on three days now.

This is completely unacceptable.

@Mark @Ram Your VPS is online. I checked a few of your sites and they successfully loaded. If you are still having problems, please contact tech support.

@Frank Your VPS guest in back online. Please let us know if you notice any problems

Well…after maxing out my resources and rebooting the server, I notice that it’s STILL not loading, so…there’s that.

@ Frank: I tested your site now and it is coming up fine. Your server is currently using about half the resources that are allocated to it.

@Brian: we are ps18686. we have sent countless emails via support. we have been down 5 days!! not even ftp or admin access. our staff cannot even work!! this is insane, and the fact that we are totally ignored… like this doesn’t even matter. what kind of service is this and why has chat been offline for 5+ days? this is the 3rd time within the past year or so our sites have been totally down for several days or a week, and our email was down last month for a whole week. has it ever crossed your mind about why you have seemingly non-stop hardware problems?

Our site was up for about 10 hours or so and now it’s back down, again with no email communication from DreamHost, nor have they replied to either of the last three emails I’ve sent in the last few days.

The CPU on our server is maxed out which is something we can’t increase or decrease, so we’re left sitting here twiddling our thumbs until someone decides to fix it, yet again.

@Frank: Your VPS is up at this time! Apologies if you have not received an email reply, as we are working through all the messages as quickly as we can. Your services are online at this time, though!

@Malcolm: Your VPS is also up and running at this time. As I mentioned above, we are working hard to make it through all the messages about this issue and I sincerely apologize if you have not received a reply yet. Know that we are working very hard to keep all VPS guests online during this restore process and take quick action any time a VPS reports offline. The server will be loady until this process is complete, though.

I NEED HELP!!!!!!!! look what you say to me:
“Hi Daniel,

I apologize, I’ve gone through all of our backup servers, and have been
unable to find backups for any of these users. While we do our best to
maintain backups for all users, we can’t guarantee their availability,
and you’ll need to restore these from your own backups. I’m very sorry
that there’s nothing more we can do for you in this matter.

Thanks,
Brandon”

Hey guys, don´t say you don´t have backup, in the post about michaelkelly you said you had the info from the old hardware, well, there has to be the information, the sites that are online, is because i reuploaded them, I NEED THE INFORMATION TO BE AVAILABLE; I HAD SVN, I HAD MORE THAN 130GB THAT NEED TO BE RESTORED, THIS IS NOT MY FAULT!!!! I MOVED TO THE VPS TRUSTING YOU TO HAVE THE OLD HARDWARE BACKED UP, I NEED THESE USERS URGENTLY!!! I DON´T KNOW WHAT YOU HAVE TO DO, BUT I DON´T HAVE BACKUP OF THESE USERS, AND THEY ARE THE MOST IMPORTANT ONES!!!! PLEASE SEND THIS TO SUPERVISOR, OR MAKE MAGIC, I DON´T KNOW, IT´S SIMPLY IMPOSSIBLE YOU DON´T HAVE THE INFORMATION IF YOU HAVE THE OLD HARDWARE, I NEED THIS TO BE FIXED AS A MATTER OF DEATH, MY CUSTOMERS ARE KILLING ME!!! A LAWYER CALLED ME THIS MORNING, PLEASE HELP!!!!!!!!!!!!!

davellanedam
intermarket
parperf
queenmaster
totumo
chiky_fly

Hi guys, i´m dying here, i just logged into michaelkelly with some usernames and the info is there, i just need you to move from michaelkelly to my VPS ps86583, i know you already did it, but when i requested a move you were in trouble, and i´m seeing some restored info there, so, please, MOVE THEM AGAIN!!

patrickm: the VPS is running, but we don’t have our files restored so we can’t operate & our staff cannot work. we’ve been unable to operate for 6 days now! any changes we make now might interfere with or get wiped out by the restore process. it would be nice if you could provide us info & a timeline for when the restore will be complete so we know we are safe to work. i can’t believe DH has not provided this detailed info to its customers.

@Malcolm: Any data you upload at this time will not be overwritten by the restore process. Our rsync process is not overwriting newer files on the server and uploading those will not interfere in any way. We cannot provide an ETA on this restore, unfortunately. We are nearing completion now, but it is very hard to say how much time remains.

@patrickm: ok – for example, a particular directory shows files that were already deleted before our site went down last Saturday & is missing all the files added from the past 2 months. if i add new files to this directory, these newly added files will remain after the restore is complete? and the files missing from the past 2 months (that were all there on Saturday) will be restored?

Could some one please take a look at : ps54088 and try to bring it back to live, thanks.

@Steban: Your VPS is now working again.

You guys are joking, right?

We’re pushing almost a week of down time and every time you say our VPS is working, it’s actually not. I try to write a post on my site and it crashes the VPS, I tell you about it and the site’s now been down for between 24 and 36 hours, and still nothing, no communication or anything.

It’s the equivalent of your car breaking down in the middle of the road and instead of your mechanic telling you what’s wrong with it he fixes it just enough so that you can drive it out of the garage and then puts his hand out still expecting payment. Except instead of only happening once, this happens Every. Single. Day. For a week.

and our existing ftp files are still inaccessible. If we try to click on folders we get “Could not retrieve file listing. Permission denied.” errors. what is the point of a VPS being up & running if we can’t access our files??! we still can’t load data and it’s been 7 days our sites have been down!! this is insanity!!

@Frank: I’m restoring your files at this time. Your VPS server is up however, the restore has yet to reach your VPS. I am manually restoring it for you and will email you when it’s complete.

@Malcolm: I just tested uploading files on your server and was able to do it without any issues. What user are you testing? Let me know and I can take a look.

@Oscar – why is it we can’t get help via the support ticket? we’ve been ignored here for 7 days w/ barely any communication. it isn’t just 1 user from what our admin said. if i go into Molly_pending -> Malcolm ALAC – P+R+C+B i get this message for any folder i open down to the letter ‘U’. we can’t run our site though regardless b/c the data is outdated by 2 months. will the restored data not show up till it’s 100% complete in the transfer?

Malcom, I responded to you, via our support system.

we’ve been down for 10 days now!! i realize the VPS is up but our files are 2 months old and anything we do now will be overwritten as it has already been proven due to files gone missing. can you please see our support ticket and reply? these important directories also have missing files:

/home/[company name]/Molly_Pending/Malcolm (for Pending page)/Malcolm ALAC -R&B
/home/[company name]/Molly_Pending/Malcolm (for Pending page)/Malcolm ALAC – HH
/home/[company name]/Malcolm_Downloads
/home/[company name]/development.[company name].com/public/itunes_artwork

and really this whole directory had updated files missing:
/home/[company name]/Malcolm_Uploads/Library B (metadata needed)

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups (RESOLVED)
7:40 pm

Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 25th, 2012 at 7:40 pm PST) by Brian@dreamhost

Due to issues with the drives on the web VPS host server ‘homie-vserver173′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver173′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. — Brian

Update Mar 17th, 4:20pm PDT: The restore is now complete for all guest machines. If you have any problems with your VPS guest or are missing data,  please let support know. – Oscar

Update Mar 17th, 6:50am PDT: All VPS guests continue to report as online and no problems with the restore process have cropped up. If you have any problems with a VPS guest please let support know. – JJ G

Update Mar 17th, 1:15am PDT: All VPS guests are reporting as online and we’re showing no problems with the restore. If you have any problems with a VPS guest please let support know. – JJ G

Update Mar 16th, 12:18pm PDT: We are showing no issues with any of the VPS guests on this host and the restore is continuing along without any trouble! Please contact support with any questions, or issues! – Brian S

Update Mar 16th, 5:15am PDT: We are showing no issues with any of the VPS guests on this host and the restore is continuing along without any trouble! Please contact support with any questions, or issues! – Chalise

Update Mar 15th, 9:50pm PDT: We are showing no issues with any of the VPS guests on this host and the restore is continuing along without any trouble! Please contact support with any questions, or issues! – Seohee

Update Mar 15th, 4:20pm PDT: We are showing no issues with any of the VPS guests on this host and the restore is moving along without any trouble! Please contact support with any questions, or issues! -Brian S

Update Mar 15th, 11:04am PDT: We are showing no issues with any of the VPS guests on this host and the restore is moving along without any trouble! Please contact support with any questions, or issues! -Brian S

Update Mar 15th, 5:20am PDT: We are showing no issues with any of the VPS guests on this host and the restore is moving along without any trouble! Please contact support with any questions, or issues! -Patrick

Update Mar 15th, 12:30am PDT: All VPS guests on this host server are reporting online and the restore is progressing without any issues. Please contact support with any questions, or issues! -Patrick

Update Mar 14th, 8:13pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Dan J

Update Mar 14th, 2:55pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Oscar

Update Mar 14th, 4:48am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Chalise

Update Mar 13th, 9:31pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Dan J

Update Mar 13th, 5:37pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Dan J

Update Mar 13th, 11:13am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Andrea

Update Mar 13th, 6:20am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Chalise

Update Mar 12th, 7:01pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Andrea

Update Mar 12th, 10:20am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Justin

Update Mar 12th, 5:00am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Chalise

Update Mar 11th, 8:03pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Andrea

Update Mar 11th, 1:45pm PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Justin

Update Mar 11th, 6:30am PDT: The restore process continues and all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -JJ G

Update Mar 11th, 1:30am PST: The restore process on this machine continues to execute and the machine remains stable. All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -JJ G

Update Mar 10th, 5:05pm PST: The restore for this host is continuing without any issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 10th, 5:39am PST: With the exception of 1 VPS, all services are online for this host. The restore is moving along still and the load is looking good on the server. Please contact support if you are experiencing any issues, or have any questions! -Patrick

Update Mar 10th, 1:10am PST: This restore is moving forward and all VPS guests are reporting online at this time. Please contact support with any questions! -Patrick

Update Mar 9th, 7:50pm PST: The restore for this host is continuing without any issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Dan J

Update Mar 9th, 2:30pm PST: The restore for this host is continuing without any issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 9th, 5:53am PST: This restore is still running well and all guests are reporting online! Please contact support if you have any questions, or are experiencing any issues! -Patrick

Update Mar 9th, 3:01am PST: This restore is still running well and all guests are reporting online! Please contact support if you have any questions, or are experiencing any issues! -Patrick

Update Mar 8th, 9:58pm PST: All VPS guests on this host server are reporting online at this time. Please contact support if you are having any issues with a VPS hosted on this server. Thanks for your continued patience! -Dan J

Update Mar 8th, 6:17pm PST: All VPS guests on this host server are reporting online at this time. Please contact support if you are having any issues with a VPS hosted on this server. Thanks for your continued patience! -Dan J

Update Mar 8th, 12:25pm PST: The restore for this host is continuing without any issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 8th, 5:45am PST: This restore continues without issue and all VPSs are reporting online at this time! Please contact support if you have any questions or concerns! -Patrick

Update Mar 8th, 3:15am PST: This restore continues without issue and all VPSs are reporting online at this time! Please contact support if you have any questions or concerns! -Patrick

Update Mar 7th, 9:51pm PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Dan J

Update Mar 7th, 5:06pm PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Seohee

Update Mar 7th, 11:30am PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 7th, 5:00am PST: The restore is continuing to run and all VPS guests on this host are up and running. If you notice any issues with your VPS on this server, please contact support.  – Chalise

Update Mar 6th, 9:46pm PST: All VPS guests on this machine are reporting online and have no apparent issues. We’ll update this post as the restore progresses. Please contact support if you are having any issues with a VPS hosted on this server. -Dan J

Update Mar 6th, 4:35pm PST: All VPS guests on this machine are reporting online and have no apparent issues. We’ll update this post as the restore progresses. Please contact support if you are having any issues with a VPS hosted on this server. -dan w

Update Mar 6th, 1:35pm PST: All VPS guests on this host server are reporting online at this time. Please contact support if you are having any issues with a VPS hosted on this server. Thanks for your continued patience! -Brian S

Update Mar 6th, 10:30am PST: All VPS guests on this host server are reporting online at this time. Please contact support if you are having any issues with a VPS hosted on this server. Thanks for your continued patience! -Justin

Update Mar 6th, 3:26am PST: All VPS guests on this host server are reporting online at this time. Please contact support if you are having any issues with a VPS hosted on this server. Thanks for your continued patience! -Patrick

Update Mar 5th 9:22pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. The restore is continues as well. If you’re having any issues with your sites, it may be unrelated, so lease contact support with the details, so we can assist you. – Dan J

Update Mar 5th 4:05pm PST: The earlier connectivity issues are no longer affecting your server, so your sites should be up again, running from backup. The restore is continues as well. If you’re having any issues with your sites, it may be unrelated, so lease contact support with the details, so we can assist you. – Andrea

Update Mar 5th 12:00pm PST: We’re currently experiencing some connectivity issues that is causing services on this machine to be unavailable. This incident is unrelated to the recent issues this server has been experiencing on its own. Please keep an eye on this status post and we will update it as we have new information.

http://www.dreamhoststatus.com/2012/03/05/connectivity-issues-to-some-of-our-datacenters

Once that issue has been resolved and the service has been fully restored, we will return to updating this post with information specific to this machine. Please contact support if you have any questions or concerns regarding this. -dan w

Update Mar 4th, 9:31pm PST: We’re continuing to keep an eye on this restore process. All VPS guests should be up, but if you need assistance with your sites, please don’t hesitate to contact support with the details. -Andrea

Update Mar 4th, 4:10pm PST: The file restore process for this machine continues and we’ve verified that all VPS guests are reporting online. If you have any issues, or questions, please contact support for assistance! – dan w

Update Mar 4th, 9:20am PST: The restore for this host continues and we’ve verified that all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! – Justin

Update Mar 3rd, 11:15pm PST: The restore for this host continues and we’ve verified that all VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -JJ G

Update Mar 3rd, 2:10pm PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 3rd, 3:45am PST: This machine remains stable and all VPS guests continue to report as online. Please let support know if you have any issues or questions. -JJ G

Update Mar 2nd, 8:20pm PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Dan J

Update Mar 2nd, 2:20pm PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Oscar

Update Mar 2nd, 5:55am PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Patrick

Update Mar 2nd, 1:35am PST: The restore for this host is continuing without any new issues! All VPS guests are reporting online at this time. If you have any issues, or questions, please contact support for assistance! -Patrick

Update Mar 1st, 5:30pm PST: The restore continues on this server and all VPS guests are still responding. If you have any questions or concerns, or notice any issues, please contact support. More updates to come as the restore progresses! – dan w

Update Mar 1st, 10:00am PST: This restore is continuing normally and all VPS guests are reporting online at this time. If you have any questions or concerns, or notice any issues, please contact support. More updates to come as the restore progresses! – Justin

Update Mar 1st, 4:36am PST: This restore is continuing normally and all VPS guests are reporting online at this time! More updates to come, but please contact support if you notice any issues with your guest! -Patrick

Update Feb 29th, 6:36pm PST: All VPS guests are up and the restore is continuing without any issues. There are a couple of down Apaches which our admins are working on now. We appreciate your patience! We’ll continue to provide you updates, so please check again soon. -Seohee

Update Feb 29th, 1:05pm PST: We were able to fix the load issue caused by the backup brick and things are running smoothly again. The restore is still processing and we’re monitoring the server to make sure it remains functional during the restore process. -dan w

Update Feb 29th, 12:10pm PST: The restore is still running on this server. The load has spiked due to an issue with the backup brick, so it has made some sites load slowly or appear down. Our admin team is currently working on reducing this load and restoring service to all the VPS’s stored on it. We hope to have things running smoothly again as soon as we possibly can. We’ll update this post when things look back to normal. -dan w

Update Feb 29th, 9:30am PST:The restore is continuing to run well, with all VPS guests running. If you have any questions or concerns, or notice any issues, please contact support. More updates to come as the restore progresses! – Justin

Update Feb 29th, 4:39am PST: All VPS guests on this host machine are up, and the restore is continuing to run smoothly. If you have any questions or concerns, or notice any issues, please contact support. More updates to come as the restore progresses! – Chalise

Update Feb 28th, 6:18pm PST: The restore is currently running and all VPSes and Apaches should be up and online . We’ll update this post with more information as we have it, but if you have any questions or concerns, please contact support with the details. – Seohee

Update Feb 28th, 9:30am PST: The restore is running normally. Our admin is still working on getting some of the last remaining guests on this host machine up and running as they are having to be manually restored.  You can upload your data now, if you’d like, as it will not be overwritten by the restore. We’ll update this post with more information as we have it, but if you have any questions or concerns, please contact support with the details. – Justin

Update Feb 28th, 5:00am PST: The restore is running normally. Our admin is still working on getting some of the guests on this host machine up and running as they are having to be manually restored, I apologize for the delay. You can upload your data now, if you’d like, as it will not be overwritten by the restore. We’ll update this post with more information as we have it, but if you have any questions or concerns, please contact support with the details. – Chalise

Update Feb 27th, 8:43pm PST: The restore has started and is running as expected. Some of the guests are reported down and an admin is working on getting them back up.  Some will have to be manually restored, which is what is causing some delays. Sorry about the continued inconvenience. If you wish, you can upload your data now, and it will not be overwritten by the restore.

We’ll update this post with any progress reports we get, but if you have any questions or concerns, please contact support with the details. – Seohee

Update Feb 27th, 10:58am PST: The restore has started and is running as expected. Some of the guests will have to be manually restored, that is what is causing some delays. Sorry about the continued inconvenience. If you wish, you can upload your data now, and it will not be overwritten by the restore.

We’ll update this post with any progress reports we get, but if you have any questions or concerns, please contact support with the details. -Andrea

Update Feb 27th, 8:30am PST: All VPS guests on this server should now be online! Our admin is now working on getting the restore running again. We will keep you posted!  – Chalise

Update Feb 27th, 8:10am PST: Many of the VPS guests on this server are now online, and our admin is continuing to work on bringing up the rest of them.  – Chalise

Update Feb 27th, 12:05am PST: Our admins are still working on bringing the VPS guests on this host machine online, and we will certainly update once that occurs; unfortunately, we do not yet  have an estimate as to how long this will take. We, again, apologize for the inconvenience this is still causing. – Chalise

Update Feb 26th, 8:26pm PST: We were able to bring the backup server online, and we’re bringing back up all the VPS guests on this host. It will take a while, but everything looks to be in order. Once the guests are up, they will mount to the backup server to serve your sites and the restore of data from backup to the new hardware will start. We’ll update this post again when we get news on the progress, but we don’t have an estimate when that may happen. Sorry about the inconvenience this is still causing. – Andrea

Update Feb 26th, 5:06pm PST: Unfortunately, the old server became usable earlier today, so we’re setting up a new one to fail the data over. We’re currently waiting on the backup server to come online, so we can mount the new server on it, and serve your sites during the restore process. Very sorry about the inconvenience this has been causing. – Andrea

Update Feb 26th, 9:30am PST: VPS services are coming back up, further updates will be posted as the situation progresses. – JJ G

Update Feb 26th, 5:15am PST: Our admins are aware of servers on this machine which are not currently running. We are working on getting services up and will soon post another update on the progress for that.  – JJ G

This entry was posted on Saturday, February 25th, 2012 at 7:40 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

75 Responses to “Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups (RESOLVED)”

The following VPS machines are affected:

ps37393 ps37419 ps37462 ps39142 ps39159 ps39162 ps39381 ps39388 ps39414 ps39418 ps39423 ps39561 ps39569 ps39584 ps39947 ps39975 ps40397 ps40685 ps40898 ps41068 ps41072 ps41422 ps41510 ps41623 ps41903 ps42001 ps42303 ps44457 ps44583 ps45065 ps45632 ps45690 ps46094 ps46106 ps46127 ps46554 ps46791 ps46956 ps47188 ps47357 ps47728 ps48057 ps48081 ps49327 ps50502 ps52416 ps52432 ps53147 ps53373 ps54114 ps55484 ps55485 ps56454 ps56458 ps57506 ps68922 ps71111

our site is not working ps22199

@bhavneet: Please refer to the status post for homie-vserver127. That’s the host machine for your VPS and it is currently being restored.

And what about those of us who aren’t even on VPS, but on shared plans?. All my sites still dead with all the files gone. Are you aware of it affecting lots of other non-vps customers?

I’m on machine ‘stampeders’

Ridiculous. We, too, are down. Have been down most of the day. http://Www.feneal.com … same song, difft day.

@Laurence: Your sites should be up now. I updated that post about an hour ago. Write to support if you are having any issues with your sites being down now.

@Oscar

Yes, it is finally up!. Though it seems to be a backup restore point from a month ago?. Not sure if you have a more recent backup restore point.

Ah well at least the sites are actually up now and i have ftp access too, so thanks :)

our site is STILL not working ie ps22199 — oscar?

Support writes me:
> Can you give me an example of what’s not working?

it’s easy:
NOTHING

EVERYTHING IS DOWN, DOWN, DOWN, DOWN….

EVERYTHING IS DOWN, DOWN, DOWN, DOWN….

EVERYTHING IS DOWN, DOWN, DOWN, DOWN……..

The Boss

In the last 48h My vps was up 7h.
This is a DISASTER.

Wow guys, impressive work.

We thought it may have been an issue with our configuration due to the site going offline at the same time we switched to ‘Dreamhost Managed’…

One of our administrators contacted your 24/7 email support team to get an answer, the least someone from this department could have done was identified our VPS was part of the ones that were affected and directed us here.

This is the 2nd weekend in a row our server has actually been taken offline due to Dreamhost… Which not only affects our website, but our credibility as a service provider to our clients resulting in a loss in revenue for us… yet I don’t see you guys offering any sort of compensation.

Once again, would like to thank you for your ‘support’.

Regards,
The Neocodex Administration Team

S.o.S .. Just returned home page .. Internal links are all inactive portal … What do I do? All pages are showing 404 errors – Http not found.

At least you got your home page back Claudio, we haven’t even got that.

Guys we need an estimate time, my VPS (ps46127) has been more than 24 hours down and I have nothing to tell my clients, who obviously are about to kill us due to the lack of information.

Less than a month ago it was the same thing, a whole weekend down. How are we supposed to do our business to pay your bills with this kind of troubles?

ps85428 not responding

You know what I love?

Chocolate.

And sodomy, of course.

February 26th, 2012 at 2:20 pmChristofer Hoff Says:

and ps8827 isn’t listed (my VPS) but all of my web properties are down; there’s no HTTPd listening on any of my domains.

I don’t see how the web/connectivity tests pass when there’s no freaking daemon listening on a port?

Update Feb 26th, 9:30am PST

Current DreamHost time: 16:31

6 hours since your last update, we’ve been out of service for over 30 hours. Our site did come online for a few moments but is extremely unstable and crashing…

Nice to see ‘consistent’ updates letting the customers know what is going on.

Very sorry about the delayed update and the downtime. We’re working on getting the backups machine online to bring up your guest servers and start the restore to the new machine. We’ll update when we have more information.

ps80498 – this is absolutely ridiculous. Site now down for two days… and counting.

@Claudio: I’m very sorry to hear it. Please note that your VPS is on HV127, and not HV173. That said, I see that your VPS is up and your site is resolving properly at this time. Please take a look and contact support if any problems persist. We’re here to help.

we got an email from support 7 hours ago that our servers are up and running. We rebooted but the site is still down 7 hours later. Our site has been down 2 whole days now. We are on ps81981. Can you please explain what’s happening here?

First time I’ve been compelled to comment here, but after 18 hours of downtime and no response at all from customer support, I’m very disapointed and frustrated to say the least.

I’ve always been impressed with customer support, despite what some people may say, but this situation isn’t good at all. Far from it. I honestly have no idea what is going on. Even tried contact through Twitter, but received the same silence…

I’m on ps27347 which hasn’t been listed, but is clearly having problems.

@Joe: Sorry to hear it. Your VPS is not on HV173, so it is unrelated to this post. I see you have a ticket open, so we will get back to you as soon as we can. We appreciate your patience!

@Christofer Hoff: Your VPS is not on HV173, and I see that your VPS and sites are up and running at this time. If you’re still having any problems, please contact support.

@Lee: I see that your VPS is on HV177, and not 173. That said, I went ahead and started up Apache on your VPS, and see that your sites are back online. I see you have several tickets open, so we’ll get back to you as soon as we can. Appreciate your patience.

@ogp: Your VPS is on HV24, not HV173. I see you have an open ticket, so we’ll get back to you as soon as possible. We appreciate your patience.

For all the others, an administrator is working on getting your VPS back up and running. We appreciate your continued patience and apologize for the trouble this has caused.

Downtime is up now over 36 hours and no way to contact our customers to let them know what is going on with all of our databases being stored on Dreamhost servers.

Are we much closer to getting our sites back online? I have no idea how this is even going to affect our standing with our subscribed customers but I can only hope they decide to stick with us.

SOS!!! Downtime is killing me! Please restore the accounts soon! we need solutions!

ps81905 Is down. I also have always been impressed with customer support, but can not get any straight answers currently, and was told it was fixed when it clearly wasn’t. We are losing advertising money now and I am losing my patience.

@SOS – please contact support with your details through the DreamHost panel. We would be happy to assist.
@Not Happy – your case is unrelated to this post, you’re on a different VPS host. I’m having someone take a look at your tickets now, we’ll get back to you shortly, but for now, it looks like you’re just out of memory.
@Neocodex – your case appears to be related to your apache service. I’m getting it looked at and will email you shortly.

Apparently we are down again. http://www.feneal.com We were up for a bit today. Can someone please get this fixed.

oh, and we are ps52432. This is ridiculous. Please, we need some immediate relief.

Well our site now seems to be up and running (albeit slowly and some of our processes are not running however we will need to look into that to see if the cause lies on our end).

@andrea, I’m not sure how the case would have related to our Apache service when we are running nginx, but I’ll leave that one for my technical team to discuss.

As mentioned by someone else on one of the other issues, along with posting this ‘message’, a notice in our control panel would be great as you guys ARE able to identify affected VPS servers. When I hit ‘reboot’ in our control panel, we simply see a ‘Success’ message after a few minutes… How can your system report ‘success’ when it is clearly down?

An automated email to clients on the affected VPS servers would also be a nice touch and a move towards better customer service as well.

All in all, our site is now accessible and we can at least get a back up of our data on the offchance it seems to crash again and for whatever reason doesn’t return.

February 27th, 2012 at 8:11 pmGaudy Rodríguez Says:

My VPS (ps46127) still down! We need a solution soon!…

February 28th, 2012 at 7:21 amGaudy Rodríguez Says:

My VPS (ps46127) still down! We need a solution soon!…

You guys are the worst. Great doing business with you.

“The restore is running normally”!!!!! The VPS has been offline for 4 days now!!!! NOTHING IS NORMAL.

You say it’s recovering and the sites are still down, you say we can upload the data but I cannot even connect through SSH, I’ve asked a thousand times when will the VPS be restored and there’s nothing but silence… this is unbelievable, so disappointing.

@Dana @Gaudy Your VPS guest back online. Sorry for any inconvenience this has caused. Please let us know if you notice any other problems.

@Hector Please provide us with the VPS guest ( psXXXXX )

February 28th, 2012 at 12:55 pmGaudy Rodríguez Says:

@brians My websites are not working (see support ticket #4778148 for examples) and i’m not able to login through SSH nor FTP. @Hector is my partner, it is the same VPS.

I echo what Hector says. “The restore is running normally! The VPS has been offline for 4 days now!!!! NOTHING IS NORMAL.”
I have never heard of such horrible web hosting business.

My clients and I are losing a lot of money. Give us our files!

February 28th, 2012 at 3:52 pmGaudy Rodríguez Says:

MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!
MY VPS (ps46127) still down! MY VPS (ps46127) still down! MY VPS (ps46127) still down!

What about ps55484? I do see my static websites, but all my WordPress sites are blown out-of-the-water.

PS46554 IS STILL DOWN!

Dear Seohee, when you write “please contact support with the details” it is just adding more insult to injury as your support has obviously completely crashed.

Starting the 4th days of downtime? Do you think you can ever rebuild your clients trust?

4th day for us too. ps52432

February 29th, 2012 at 12:02 pmGaudy Rodríguez Says:

My websites are down again…

Please check inmediately, the service was working the last 16 hours after the four days failure.

we are up, but barely. very slow.

AND NOW, WE ARE NOT UP.

PS41623 is down again. It was up early this morning.

Sorry, wrong thread.

Site was up early this morning, but is down now. ps41623

ps52432 is down still.

I NEED HELP!!!!!!!! look what you say to me:
“Hi Daniel,

I apologize, I’ve gone through all of our backup servers, and have been
unable to find backups for any of these users. While we do our best to
maintain backups for all users, we can’t guarantee their availability,
and you’ll need to restore these from your own backups. I’m very sorry
that there’s nothing more we can do for you in this matter.

Thanks,
Brandon”

Hey guys, don´t say you don´t have backup, in the post about michaelkelly you said you had the info from the old hardware, well, there has to be the information, the sites that are online, is because i reuploaded them, I NEED THE INFORMATION TO BE AVAILABLE; I HAD SVN, I HAD MORE THAN 130GB THAT NEED TO BE RESTORED, THIS IS NOT MY FAULT!!!! I MOVED TO THE VPS TRUSTING YOU TO HAVE THE OLD HARDWARE BACKED UP, I NEED THESE USERS URGENTLY!!! I DON´T KNOW WHAT YOU HAVE TO DO, BUT I DON´T HAVE BACKUP OF THESE USERS, AND THEY ARE THE MOST IMPORTANT ONES!!!! PLEASE SEND THIS TO SUPERVISOR, OR MAKE MAGIC, I DON´T KNOW, IT´S SIMPLY IMPOSSIBLE YOU DON´T HAVE THE INFORMATION IF YOU HAVE THE OLD HARDWARE, I NEED THIS TO BE FIXED AS A MATTER OF DEATH, MY CUSTOMERS ARE KILLING ME!!! A LAWYER CALLED ME THIS MORNING, PLEASE HELP!!!!!!!!!!!!!

davellanedam
intermarket
parperf
queenmaster
totumo
chiky_fly

@Justin … we’ve had no update since 10:00 PST, and our website has been down all day long. I logged 2 tickets this morning, and again they were “moved” by Dreamhost at 12:00 CST, but nothing. Not one peep from you all. What are we supposed to do? What are we supposed to think?

shhh … I hear crickets.

@Dana your server should be online now, I just started it up again

Could some one take a look at : ps54088 and try to bring it back to live, thanks.

@Steban: Your VPS is not hosted on this host machine, but I’m working on it.

@Dan J: my website is still down.

@Andrea: ps52432 is still down. Will we ever see it again?

What are you guys doing? I’ve done extensive work with VMWare in the past, and a VM should NEVER be down like this. Ever. If a physical machine dies, the VMs should be seamlessly taken over by another physical server. Several years ago, we lost one physical server and everything ran seamlessly for a week until we were able to restore it. None of our users or visitors even realized that one of our physical servers was offline. The data storage should have the same level of redundancy. Even if you lose two hard drives in separate redundant arrays, your system should be able to easily use the missing pieces from the second array, and seamlessly restore data when a hard drive is replaced, with no down time.

As of now, my personal sites have all been completely offline for over a week (and are STILL offline). If this was my business or one of my client’s VPSs (I feel sick just thinking about this having happened to one of my ecommerce clients’ sites), I’d be out of business – no ifs, ands, or buts. Add to that the numerous (though less catastrophic) problems with my business VPS (a different physical server), including one database that magically disappeared… I just can’t trust you any more. Most alarmingly, however, is that you insist that said database was apparently deleted by time travellers. We were actively using up to a week or so ago, and yet you insist that it was deleted over a month earlier. I’m sorry, but that’s outrageous on more levels than I can count.

I cannot stress enough how horrific these experiences have been. Had I not been keeping obsessive backups, had I not been maintaining separate virtual server accounts with another hosting service, I’d be supremely screwed. A VPS should always be more stable and more reliable than a single dedicated server… and you guys blew it hard. I used to love your service, but in the last two weeks you have caused me more sleepless nights and gray hairs (in one case, during a badly needed vacation) than any client I’ve had in years. As soon as I have found a suitable replacement, I’m moving all my personal, business, and client accounts off of Dreamhost.

You guys really, really, REALLY broke my heart.

Sites were up for a short period last night, and now they’re gone gone gone, AND this time, the dreamhost panel went with it, so that we can’t even attempt to restart our VPS and more importantly we can’t even annoy you with yet another support ticket. ps41623

@Matt VanAndel … REALLY, REALLY BLEW IT.

@Dan W. we really, really need our files, the sooner the better. This ticket is the oldest in your pile. Please direct all your attention to 173. We’ve been down forever!

Now that you guys appear to have gotten feneal.com restored, can you work on restoring the rest of the VPS for user riceboyler? I’m not sure how you can restore just one site out of the whole VPS… Something is VERY fishy here.

@Jason I spot-checked a few of your sites ( ex. playfulpicasso.com ) and they successfully loaded. Please let us know if the problems persists. If so, could you please submit a support request and provide the details?

Due to the ticket system is flooded. My TIME CRITICAL TICKET about a domian CANCELLATION ISSUE is being ignored. Please help !!! This is REALLY TIME CRITICAL! Thank you.

The TIME CRITICAL ticket: #4799173

@Neo – your issue is unrelated to this host machine and needs to be handled specifically by our Abuse/Security team. I see your tickets are in their queue and they will get back to you as soon as possible.

Andrea, Tried to use an old support ticket to get through… the panel is still down…
tracking # is 4813370

@anonymical – It looks like your ticket has already been addressed by our Support staff. If you have any further questions, please feel free to reply to the ticket directly so we can help.

Hi Dreamhost

How the hell does a restore take so long, please break it down for me like I am small child, hold my hand and walk me through it?

I simply cannot fathom how it would take this long, or how a disaster recovery situation like this is even remotely acceptable?

I am looking beyond stupid to all my clients, I am considering leaving and the prospect of moving numerous domains and a WordPress Multisite (along with a host of other WordPress sites) is as about exciting as vigorously applying a rusty grater to my face.

When will my VPS (48081) be running normally, when? Why am I paying this month is your service has been atrocious? Why is there no information about why they hell this is taking so long? Surely you could recover from an earthquake faster than you are recovering from the failure of a RAID controller? Pardon my ignorance but why why why?

Serious pissed at the crap service I am suddenly getting after five loyal years…

Right there with you, Zane!

I wrote up my experiences on their forum (since I can’t access my blog) to try to get some attention after receiving a form letter to my latest ticket to support informing them that no, everything is NOT moving along without any trouble:

http://discussion.dreamhost.com/thread-134322-post-151466.html#pid151466

At least it’s not just me. So much for company loyalty.

@Marius Sorry for the problems that you are experiencing. To better assist you, could you please provide us with your VPS guest ( psXXXXX )? We will gladly take a look at your specific issue.

Absolutely: ps56458. Performance seems to have improved somewhat this morning (they load!) so I’m hoping it stays this way.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups ( RESOLVED )
11:41 pm

Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups ( RESOLVED )

Posted (February 23rd, 2012 at 11:41 pm PST) by chalise

Due to issues with the RAID controller on the web VPS host ‘homie-vserver127′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host ‘homie-vserver127′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 28th, 1:23pm PST: We have confirmation that the restore to the new hardware has finished. All sites should be up. As we used the last available backups for he restore, it’s possible that you may not see the same data that was on the old hardware. If that’s the case and you need newer data but don’t have your own backups, please contact support with the details, so we can assist you. -Brian S

Update Feb 28th, 9:30am PST: The restore is currently paused but will continue shortly, but vps should be running normally from backups. There are only a couple guests left that still require attention from our admins, and will be taken care of shortly. If you notice any problems with your VPS on this machine, please contact support so we can help. – Justin

Update Feb 27th, 4:00am PST: Most guests on this server are up and running and the restore process continues, however, there are a few VPS guests we’re having a few issues with that our admin is looking into. If you notice any problems with your VPS on this machine, please contact support so we can help. – Chalise.

Update Feb 27th, 8:38pm PST: All guests on this server are up and running at this time. The restore is still running smoothly and we’re keeping a close eye on things. If you notice any problems with your VPS on this machine, please contact us to let us know. -Seohee

Update Feb 27th, 6:02pm PST: We believe that all guests on this server are up and running and have answered almost all support related to this server. The restore is still running smoothly and we’re keeping a close eye on things. If you notice any problems with your VPS on this machine, please contact us to let us know. -dan w

Update Feb 27th, 1:02pm PST: Most VPS guests on this host machine are up and running, and we are working toward bringing any remaining guests back online as well. More updates to come soon! -Brian S

Update Feb 27th, 8:15am PST: Most VPS guests on this host machine are up and running, and we are working toward bringing any remaining guests back online as well. More updates to come soon! – Chalise

Update Feb 27th,12:20am PST: Our admin team is still working to get some of the VPS guests on this host machine running again. We apologize for the inconvenience and will keep you posted with further news as it becomes available.   – Chalise

Update Feb 26th, 6:25pm PST: We’re still seeing some VPS guests not functioning correctly, but our Systems Engineer team is aware of these and are working on getting them back online. We apologize for the delayed update and the inconvenience this has been causing. – Andrea

Update Feb 26th, 4:25am PST: We are showing several guest machines as not currently running. This is something that we are aware of and it is being worked on. We will continue to post further updates regarding these issues.  – JJ G

Update Feb 25th, 6:30pm PST: Our admin team determined that the original hardware for this server needed to be replaced.  A new machine was setup and backups were mounted via NFS. At this time all VPS servers are up and running off  backups mounted over NFS. The restore is progressing without any issues. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware.  – Oscar

Update Feb 25th, 6:00am PST: The file system check on the original hardware is currently on pass 1B. Further updates will continue to be posted. -JJ G

Update Feb 25th, 3:30am PST: Unfortunately this post is being re-opened as the disk array began to give errors. The problem drive has been removed from the array and another file system check is being executed. VPS guests are being mounted on a new machine and services are coming back up. Further updates on this issue will be posted. -JJ G

Update Feb 24th, 8:40am PST: Our Admin team has gotten the original machine back in working order. As the original hardware is now running without issue, guests have been started back up and don’t need to be moved to new hardware. All affected VPS guests should now be online, but please Contact Support if you’re still experiencing issues. -Brandon

Update Feb 24th, 8:40am PST: Our Admin team has gotten the original machine back in working order. As the original hardware is now running without issue, guests have been started back up and don’t need to be moved to new hardware. All affected VPS guests should now be online, but please Contact Support if you’re still experiencing issues. -Brandon

Update Feb 24th, 6:01am PST: Our Admin team is still working on getting all the VPS guests created on the new hardware, but the restore is underway. If your VPS appears to be down, that will be resolved shortly, as the guests are being created it is just taking a little longer than usual. Please Contact Support with any questions, as always! -Patrick

Update Feb 24th,3:20am PST: Our admin team is currently working to bring up the VPS guests on the new hardware and get the restore running, but are experiencing a few issues with our backup brick. We will update again shortly with more news. – Chalise.

This entry was posted on Thursday, February 23rd, 2012 at 11:41 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

112 Responses to “Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups ( RESOLVED )”

Affected VPS guests below:
ps19707 ps19748 ps19807 ps19854 ps19858 ps20104 ps20823 ps20844 ps21688 ps21925 ps22003 ps22161 ps22199 ps22216 ps22549 ps23311 ps23652 ps23758 ps23802 ps24183 ps24229 ps25283 ps25733 ps25841 ps27064 ps28953 ps29921 ps29982 ps29986 ps30330 ps30370 ps30927 ps34717 ps34743 ps34813 ps35070 ps35085 ps35304 ps36090 ps37081 ps37114 ps38327 ps41129 ps42706 ps44783 ps44785 ps46156 ps54413 ps76340 ps80498 ps82699 ps83949 ps85492 ps85836 ps85842

Oh, wow, given that my VPS has been up and down all day, I assumed I was part of the vserver131 issue. Turns out I am part of a whole new issue, which means I probably get to wait that much longer until service is restored.

Thanks!

Just as a heads up to anyone on these servers. Don’t expect to be back up very quickly, and budget for quite a few hours of down time (I’m already 16 hours and counting on another server). If you get the chance, move your files and change your DNS to point at another host.

Read the other threads on this board to see how long you might have to wait as this issue seems to have affected a lot of servers as well over the past few days.

You’re getting me crazy.
That’s incredible how many issues in these months.

You’re getting me crazy.
That’s incredible how many issues in these months.

Affected VPS guests below: ps21925

My VPS continues to be absent from these lists of affected servers, but it still goes down when the others do. It’s ps29921 and has been down since at least 9PM PST yesterday.

Guys please tell me… it always that way?
I can’t believe -_-

Normally it’s not like this for me. Overall my experience with Dreamhost has been a good one. My only real complaint is their lack of transparency when there is an issue. I wish they were a little quicker and more straight forward about the issues and easier to get a hold of when there is one. Still, my experience has overall been good.

@Claudio Crispim: I am showing that your server is online at this time, but is rather active. Please contact Support for assistance with this particular issue, as it does not appear related to the restore.

@Steve: Please see the first comment above and you will find your server listed under the affected VPS guests. I am working with an Admin right now on your VPS guest and will be in touch via email!

@patrickm You’re right. I must have missed it in the glance over. Glad to hear you’re working on the issue for me server. Thanks.

Is PS19963 in the list of VPS? Our sites are still not online.

This has been going on for a while now. Is there going to be some kind of compensation? I have ps37081 as my server.
Would really appreciate some more details on what is going on.

Just saw the update. Nothing has changed for me. My sites are still down. Contacted support to let them know.

My site is in a messy state; lost original formatting. I contacted support last night to let them know. No reply yet.

Been checking since update went up.. Still down, but I will wait a few more before complaining.

My affected VPS (ps34813) is still offline, despite the update saying otherwise.

My original support ticket opened more than 18 hours ago is still unanswered and you suggest I open up a second one? That’s lovely.

Looking forward to another day of not being able to use my VPS.

It’s becoming a nightmare.

> All affected VPS guests should now be online,

ps19807 is still down (after more than 13 hours of downtime). Support chat is offline and nobody responds to “Contact Support” tickets.

This is getting ridiculous. This is the third serious downtime issue for the last 2 months – and DreamHost does not seem to know how to communicate the resolution of those issues except the occasional “I am really sorry about this” response.

Quote “angry user” becasue it’s the same for me:

My VPS is still down (after more than 13 hours of downtime).
Support chat is offline and nobody responds to “Contact Support” tickets!!

This is absurd.

Can we expect this problem to be resolved within next couple of hours ? Or should I purchase another PS hosting, recofigure it and push our site to it ?

We have over 70 angry users which cannot used the stuff that they purchased, because the site is down …

I am up now. Thank you.

ps19807 seems to be operational now. We started to process the complains from our users and create manually license files for them.

I am really sorry, but I cannot say “thank you”. 13 hours ago our file system was visible and available in read only mode, it should have taken only 5 min to copy the whole site configuration to another machine …

@Bram @Yona @Ciro @Angry User @ Randy Your VPS is online. Please let us know if you notice any problems.

@Dale I will respond to your support request.

@Rick Would you mind providing us with your VPS ( psXXXX )?

@Ciro Would you mind providing us with your VPS ( psXXXX )?

PS80572 seems to be down – all my sites dont load….

ps55331 also down!

Dreamhost has been wonderful for the past 2 years – with the exception of the last months – TONS of downtime, email issues….I hope this doesn’t continue.

ps80572 is definitely DOWN…again.

ps19858.. I am up for about an hour and a half.

ps25602 is down again. We had almost SIX days of uptime. Wow, Dreamhost, just wow.

@Maureen @CJ @Nathan Your VPS is online. I spot-checked a few of the domains on your account. Each domain successfully loaded. If you are still having problems, please contact Tech Support

My site is coming up very slow or not at all and navigating is almost undoable. ps41443. My server is not on any of these lists as having problems.

My site on ps41443 barely loading or not and not on any of the lists.

ps19807 seems to be down again.

We were just recovering from the yesterday mess, when we got disconnected and now the server is not accessible via ssh / http / ping.

I couldn’t even imagine such a scenario …

DOWN AGAIN
I can’t believe, I want to cry.
Dream? That’s a nightmare.

ps82699 – (DOWN AGAIN)
I’m tired.

February 25th, 2012 at 2:47 amSuper Angry User Says:

ps19807 got up again. New downtime around 1 hour and 40 minutes.

After we sshed to it, we found that it generates hundreds of errors because it contains older codebase (from 3 jan 2012 – 50 days ago). We were able to push the most recent files and site seems to work correctly now (except that it is very slow).

February 25th, 2012 at 2:53 amSuper Angry User Says:

Just for the record – during this downtime (and apparently restore of older version of the site)
* there was no official announcement
* support chat was offline
* there were no responses on e-mails sent via “Contact Support” link

We can only hope that nobody will override our site with older version of the source files …

February 25th, 2012 at 2:59 amAngry Angry User Says:

ps19807 is unreachable … again.

My boss is pissed.

Hey super Angry… just to underline:

* there was no official announcement (AS ALWAYS)
* support chat was offline (IT’S ALWAYS OFFLINE FROM THE FIRST DOWNTIME 2 MONTHS AGO)
* there were no responses on e-mails sent via “Contact Support” link (…. NO WORDS)

My Premium users want to kill me.

@davidc: Your server is not affected by this outage, however if you are still having trouble please contact our support team.

https://panel.dreamhost.com/index.cgi?tree=support.msg&amp;

@Ciro Sannino: I’ve sent you a message through the support system, your VPS is definitely being affected at this time.

@Angry User: We ran into an issue with the disk array and our admins have begun bringing services back up on another machine. Your VPS is currently up and responding but you should get better performance once the issue is resolved and you are no longer running off backups. It should also be noted that files newer then the backup source will not be overwritten.

ummm.. this is costing me money now.

This is costing money and reputation.
It’s better I don’t think about.

I opened a ticked,
waiting a reply about

1) there is a time estimate to have the VPS up again?
2) there is reasonable cause to hope that the problems are finally over?

All my premium members are paying to not access the website.
Who will refound them these days?

February 25th, 2012 at 9:05 amVery Very Angry Used Says:

After we finally configured our site with the most recent codebase, it was online for several hours and is now back down. This is the forth downtime for today.

This is RIDICULOUS.

We need THIS PROBLEM TO BE FIXED. NOW.

And what about all campaigns I ran these weeks???
Will be my audince confident in my service when the VPS will be up again?

I lost money, a lot of money.

February 25th, 2012 at 9:08 amVery Very Angry User Says:

> Your VPS is currently up and responding but you
> should get better performance once the issue is
> resolved and you are no longer running off backups.

ok here is the story:
* our site went down
* 2 hours later our site went up with codebase from 3 jan 2012 (and some additionally missing files)
* we spend a couple of hours to identify and fix all the issue (deployed the latest code)
* now the site is again down

GUYS, PLEASE STOP ***ING WITH OUR SYSTEM

People who subscribed recently are writing me:
they think my website is a FRAUD! CAN YOU UNDERSTAND?

Is this a nightmare?! WAKE ME UP PLEASE!
In 2 days I lost around 800$, and people think I’m a fraud…
GREAT

Is ps29921 part of this latest batch of problems? It was part of the first group and my sites are down again.

My sites are up, but I can’t access the FTP to upload new files to the server. Could this be related?. Contacted support but no response so far.

Actually scratch that, my site is down. well the index page is up, but the other pages are no longer there. grrrrr

I opened a ticked,
waiting a reply about:

1) there is a time estimate to have the VPS up again?
2) there is reasonable cause to hope that the problems are finally over?

I’m losing MONEY and especially TRUST.
May I have at least an answer?

Wow, ok, now i visited all my websites and it says Index of/ and lists two files

favicon.gif 25-Feb-2012 10:14 0
favicon.ico 25-Feb-2012 10:14 0

I hope you goes have redundancy backup on this. Dear oh dear…

Me too:

favicon.gif 24-Feb-2012 21:05 0
favicon.ico 24-Feb-2012 21:05 0

ANSWER PLEASE.
I’m losing MONEY and especially TRUST.
May I have at least an answer?

So difficult to have updates?

jabber service for estudobiblico.org —– error — Your jabber server is missing its configuration file.

Hey. My site is unstable .. repeatedly has problems, that are not resolved .. With that lose money, which is not good … What are you doing to solve my problem? If persisir, I’ll have to find another host .. You are forcing me to make a decision that I have been putting off .. What shall we do?

My VPS came back up earlier this morning but it must have been restored from backups because all of the changes I made in the past 24 hours have been reverted to previous copies.

Thankfully, I’ve already started moving all of my stuff off of the Dreamhost VPS service and will be canceling soon. It’s a joke.

Actually, forget previous copies. My entire VPS was restored with data from 02-14-2012.

That’s amazing.

Again, thankful I spent the time after yesterday’s outage getting almost everything off the VPS and moved elsewhere.

Thing is I’m not even on VPS but shared, does this hardware fail only affect vps or shared users?.

Either way all my sites are still down, with all the files missing, and no response from customer support after 2 hours… and counting :(

February 25th, 2012 at 12:41 pmMega Angry User Says:

> @Angry User: We ran into an issue with the disk array and our admins have
> begun bringing services back up on another machine. Your VPS is currently up and
> responding but you should get better performance once the issue is resolved and
> you are no longer running off backups.

I suppose that by “better performance” you meant – zero load of the server because users cannot access it.

Latest downtime continues for more then four hours, without any information from DreamHost what’s going on and when the problems will be fixed.

This is the worst joke I have ever seen.

Looks like we r down again, too. Both website and email. http://Www.feneal.com

February 25th, 2012 at 1:58 pmDream(???)host Says:

YOU ARE CRAAAAAAZYYYYY!!!!
People are losing money… included me.
who will refound me? you will?

TOTAL DISAPPOINTED
aaaarrrgggg

IS this why my site was redirecting and is now offline?

Now 5 hours downtime, no response from customer support, no status update here, all sites dead with 0.0 disk usage.

Funny thing is I’m usually the first to defend Dreamhost and point out that all web hosters out there suffer problems from time to time.

Must admit, given everything thats happened the past couple months…. i’m having some serious doubts…..

Now my site just says, “It works!” I disagree.

You guys are down without customer service way too often these days. I will be transferring all my site once I have access regain access to back everything up! I guess you do get what you pay for…

I’m sick to read “sorry-apologize-sorry”.
I WANT EXPLAINATIONS.

pathetic

Our site is on VPS too and has been down for many hours. When is this going to be resolved??

Thanks for the 6:30 PM update but my VPS isn’t online (ps34813) despite what the update says.

It’s been down (most recently) for almost six hours straight.

Guys my VPS (ps46127) is still down, I also sent a support ticket. Whats’s wrong?

February 25th, 2012 at 8:23 pmTired Angry User Says:

ps19807 is still down after 12 hours.

My English is too bad to express my feelings.

> Update Feb 25th, 6:30pm PST:
> At this time all VPS servers are
> up and running off backups mounted
> over NFS. The restore is progressing
> without any issues.

That is pure BS – nothing is up and running.

Same was true previous 3 times when you said that problem was resolved.

> If you are seeing any issues with your data,

We have been seeing issues with our for last 72 hours,
nothing has changed.

> please contact support so we can assist in restoring it from the old hardware.

We did try to contact support several times, no response at all.

Dreamhost use to be a dream! I want to stand by you guys but 2012 hasn’t been good at all! What’s up?

February 25th, 2012 at 10:17 pmTired Angry User Says:

ps19807 got up – I can ping it and ssh to it, but file system permissions are totally messed, and I cannot enter the web directory and check what’s the situation there.

Half of the systems does not seem to work.

Can somebody please provide us with information re what exactly was done and what are the possible problems and side effects that we need to be aware of ?

February 25th, 2012 at 10:25 pmTired Angry User Says:

Although some parts of the system seem to work, we cannot get in control of our server and start to work on getting it back fully online.

Reason: 1. File permissions are messed and 2. We cannot logon as root

xxx@ps19807:/home$ sudo su –
sudo: must be setuid root

Still no response from support tickets.

February 25th, 2012 at 10:26 pmTired Angry User Says:

Do I need to create twitter account and start spamming the personal CEO twitter stream in order to get some basic info about what’s the situation with our PS host and how we can get access to it ?

February 25th, 2012 at 11:20 pmTired Angry User Says:

I cannot even dump the myslq database and move it somewhere else – because of the permissions problems, I am not allowed to write anywhere on the file system …

February 26th, 2012 at 2:03 amTired Angry User Says:

Our server is accessible via ssh, but is totally messed because of permissions and timestamp problems.

We have almost 20 hours of downtime and there is no indication when the system will work again correctly.

Please give us back our root privileges, so at least we can fix the mess ourselves.

All those problems have been communication via “Customer Service” link – without any response from support team.

We finally got response from member of support team who fixes some of the permission problems and enable sudo.

We were able to regain root privileges and clean up the mess (one of the **** problems was that /dev/null was not writable for everybody).

System seems to be up and running relatively fine right now (we are still checking).

Summary:
* Total downtime: around 40 hours (20+ for the last downtime)
* Man power wasted: 6 man/days (during the weekend)
* Unhappy users (who cannot activate the legally purchased software): almost 200

I have never imagined that we may stumble upon such problems … apparently we need to redefine our disaster recovery policy (and possibly look for another hosting solution).

@Ciro: One of our managers has replied to your support issues. Please reply to that email with any further problems.

@Tired Angry: I’ve replied to all your support tickets and will be available for any further issues you may have, just reply to any of my messages.

@Brian: Sorry that looks like there was a missing config. Your site is responding now.

@ogp: Your services are not hosted on this machine. I’m contacting you through the support system.

oh yes… this is the reply 8h ago :
<>

Everything perfect…
JUST PITY MY WEBSITE IS DOWN.

I also replied to this ticket 8h ago.
No answers.

oh yes… this is the reply 8h ago :
“Great news! I manually restored all of your files so you should be all set now”

Everything perfect…
…………………JUST PITY MY WEBSITE IS DOWN.

I also replied to this ticket 8h ago.
No answers.

Dear Simon:

When I “Imagine the Web, My Way” it does not include frequent downtime, uninformative excuses, inability to restore service in a timely manner, and sloppy restore operations that leave major and different gaps in the restored configuration every time it happens.

When I “Imagine the Web, My Way”, it does not include recovery operations that leave the server in a series of random states over a period of several days, with no useful communication before, during, or after as to what is happening.

When I “Imagine the Web, My Way”, it does not include the hidden cost of needing to do such recovery operations several times per year because someone at your end isn’t doing their job right in the first place. I’m up to four of these episodes in the trailing 12 months.

When I “Imagine the Web, My Way”, it has all the services you offer. But they work. And when they don’t, the fix is quick and sure.

When I “Imagine the Web, My Way”, it does not enable operational errors made by someone in LA to become emotional heartburn for me.

Sorry, Simon, it isn’t working out for us because you have operational issues. Guess it’s time to say so long.

February 26th, 2012 at 6:14 amThe Angry User Says:

> jj Says:

> @Tired Angry: I’ve replied to
> all your support tickets and …

jj, personal thanks for your help to clean up the mess on our system.

I’d rather not share the feelings in my company about the general situation with Dreamhost.

Take care.

What do you think to update?

You can have issues of course, as everyone…

But
YOUR MISSING UPDATES
YOUR NO ESTIMATION TIME TO END

IS INTOLERABLE.

I GOT THIS MAIL:

“Hello, He went ahead and fixed it and your sites are now loading.”

ARE YOU KIDDING ME?
My chache browser is clear
and if I go to http://ftp.cg-blog.com with filezilla… there’s no files inside!!!
Also vps status is like a dead man.

How is it possible my blog is active?

*sigh*

ps34813 is still offline. Has been since 2PM on the 25th, despite the 6:30 PM update posted here.

Support was contacted (#4774116) with the highest crit level.

STILL NO RESPONSE. AND NO VPS.

ALL MY FOLDERS ARE EMPTY!!!

designkeeper.com / cg-blog.com / cgcertify.com

I can check them with Filezilla.
REPLY PLEASE.

DEAR FRIENDS,
ALLA MY FOLDERS ARE EMPTY and here the message I got right now:

” Due to the hardware issues with your VPS’s host machine
(homie-vserver127), we were only able to retrieve your data from the
backups servers.

Unfortunately, the backup server did not contain your most current data,
so all the data that is currently on your server is all we were able to
find.”

This is a disaster.

DEAR FRIENDS,
ALL MY FOLDERS ARE EMPTY and here the message I got right now:

”Due to the hardware issues with your VPS’s host machine
(homie-vserver127), we were only able to retrieve your data from the
backups servers.

Unfortunately, the backup server did not contain your most current data,
so all the data that is currently on your server is all we were able to
find.”
This is a disaster.

I LOST ALL MY DATAS……. and you?

It’s official DREAMHOST lost my datas
and they haven’t backup even if I was paying for backup service.

ps29921 is still offline. It has been almost thirty-six hours of on and off downtime over the past two and a half days. I received a message that my problem was moved to our severe problem queue, but that was fourteen hours ago. Do we have any sort of time frame for ps29921?

At least you guys are getting support responses. My highest crit ticket has been open for more than 15 hours now. Not a single response. My VPS is still down, coming up on 24 hours straight.

You would think that, given that they know which VPS servers are affected, they would have reached out to the customers directly, instead of us having to refresh this status page to figure out what is going on with our servers.

@Ciro: I’m very sorry to hear that you’ve run into issues. However, I am seeing that your site is resolving properly and your site files are on the server. If you still see any problems, please do contact support@dreamhost.com. We’ll do all we can to help.

we got an email from support 7 hours ago that our servers are up and running. We rebooted but the site is still down 7 hours later. Our site has been down 2 whole days now. We are on ps81981. Can you please explain what’s happening here?

I realise you guys are still dealing with these issues however for those who have their VPS’s running, it would be good if you allowed us to re-upload our sites from offline files rather than having all of our directories set as being owned by the root user and thus unchangeable.

In my case, I see that you’ve got a ~month old backup of my VPS in the home.restoring directory on my vps (ps44783) but it would be so much better if I could just re-upload the files from my offline backup which is up-to-date and at least take a little bit of the pressure off you guys instead of waiting for you to do something.

Also when these sorts of problems occur it would be far more professional of your support staff to be fielding those affected as a matter of priority rather than ignoring and not responding to support tickets.

An administrator is aware of the VPSes that are down and is working diligently to get them back up and running as fast as possible. We appreciate your continued patience and apologize for the inconvenience this has caused.

@Seohee OMG I CAN’T BELIEVE! the vps is down again???!!
BASTAAAAAAAAAA!!!!!!!

ps19807 is unaccessible again (via http and ssh)

DreamHost should change it’s name to JokeHost

ps37081 is still down. 4 days and counting.
This is outrageous!

February 27th, 2012 at 4:14 amMega Angry User Says:

ps19807

At the moment I am logged on the server and I see
that some files on the live system are replaced
WITH OLDER FILES

PLEASE STOP THIS PROCESS IMMEDIATELY.

WE HAVE REQUESTED NUMEROUS TIMES THAT YOU DO NOT
PERFORM ANY MODIFICATIONS ON OUR SYSTEM WITHOUT
OUR EXPLICIT CONFIRMATION.

Guys, you are a f**ng joke !

February 27th, 2012 at 4:19 amMega Angry User Says:

This is horrible, horrible, horrible, horrible way to threat the data of the users that use your service.

We are setting up a task to upload the latest codebase every 1 minute.

We are downloading all our databases and will start to looking for alternative hosting vendors tomorrow. (The switch will be slow and painful, but leaving with Dreamhost does not seem to be a better option.)

They did the same thing to my servers five days ago, and I’m still waiting on one of them to work properly. I made the switch to site5 hosting, and I’m very impressed with their customer support. You can even get cloud VPS that has full redundancy on your VPS servers. Their livechat people are very knowledgable, and the tickets I entered for help setting up SVN, configuring DNS, etc. were all fully resolved within 30 minutes.

http://www.site5.com
http://www.site5.com/in.php?id=127227 (my referral link if you want to use that).

@Bram I’m very sorry that you were still experiencing issues with your VPS. I’ve verified that it’s now online and fully operational.

@Angry User / Mega Angry User Unfortunately when there is a machine-wide hardware issue, we don’t have an easy way of determining whether you’d like the files restored or not. They should only be restored once, and won’t be restored cyclically.

VPS has been pingable all morning but I’m unable to log in.

I’m going to assume that work is still being done on ps34813, since support has assured me that they would contact me directly once the issue is resolved.

Coming up on 48 hours of straight downtime and extremely intermittent service for more than four days.

February 28th, 2012 at 12:35 amTired Angry Users Says:

@Brandon

Sorry but this is BS.

> Unfortunately when there is a machine-wide
> hardware issue,

Yes, I know that there has been hardware issues (and I can live with them), but what I was is pure incompetence dealing with “software configurations” & “backups” in a period of 5 days.

Our server was going offline and online – and every time when we saw it online it contained a different set of files. And every time when we fixed all the problems, deployed the latest codebase and made the system relatively stable, DreamHost succeeded to ruin it in some other way. Even after we asked you multiple times to stop making any changes on our system.

> we don’t have an easy way of determining
> whether you’d like the files restored or not.

You are saying that there is no easy way to determine if we WILL GO MAD if you DELETE IMPORTANT NEWLY DEPLOYED FILES from our system ? And that’s why you decided to just delete those files (pretending that you “restore files from backup”) ?

That’s pathetic.

Since the first time DreamHost rendered our system unusable, we have requested multiple times that YOU DO NOT TOUCH IT any more without explicit approval for us – and this was confirm by support team member “jj”

> (Please DO NO MAKE ANY CHANGES on our servers, before you get our
> confirmation)

jj> I’ve added a note to your account making with that statement as well.

Guess what ? DreamHost succeeded to damage the system three more times after that discussion.

> They should only be restored once, and won’t be restored cyclically.

Lies, damned lies, and comments from Dream Host team.

Can you imagine what is my most recent experience with ps19807 ?

Yesterday evening, we decided to stop the periodic rsync task (stupid us) and can you guess what happened ?

When I woke up this morning, I found that DreamHost has deleted – again – various important files (new downloads which were announced the the community), and has overwritten important configuration file causing the system to be down for another 7 hours.

DreamHost also succeeded to delete all log files created during last couple of days which may have helped us to slowly recover from the mess …

I wonder what we can expect today, it’s only Tuesday after all …

(And I still expect to hear what Simon Anderson has to say about this)

ok here is the good news !

!!! DreamHost didn’t delete any of our new files during the last 7 hours and 20 minutes !!!

!!! We are accepting bets on what problems you think will appear under our account within next 24 hours !!!

I canceled my VPS service over this outage and the way it was handled.

My data was never fully restored (missing entire directories) and after I was able to gain access and make changes, my system appeared to revert itself back to a different set of data, which included overwriting of changed files and restoration of deleted directories.

I got a few “sorry for the inconvenience” from support and I doubt my cancellation will raise any awareness about what they did wrong.

They chased away a loyal Dreamhost customer and with my VPS safely located elsewhere, I am now shopping around for a place to move my shared hosting.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘stampeders’ restoring from backups (RESOLVED)
12:28 pm

Shared web server ‘stampeders’ restoring from backups (RESOLVED)

Posted (February 23rd, 2012 at 12:28 pm PST) by brians

Due to corrupt operating system data on the shared web server ‘stampeders’, we have moved the server on to new hardware. We are restoring all data from our backup server to this new machine. At this time, only operating system data is affected, all customer data shouldn’t be affected at all. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you notice any problems, please let us know via our Support system under Contact Support, in your panel.

Please note, this is regarding the shared web server ‘stampeders’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 2nd, 1:50pm PST: Great news! The restore has now finished on this server! If you are missing any data at this time, please contact support and we can check the backups for you  and restore any missing data. – Oscar

Update Mar 2nd, 4:55am PST: The restore for this server is still moving along without issue. As we previously mentioned, the load is still artificially high on the server due to the NFS mounted backups but all apaches are running at this time. We sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support for assistance! -Patrick

Update Mar 2nd, 1:37am PST: The restore for this server is still moving along without issue. As we previously mentioned, the load is still artificially high on the server due to the NFS mounted backups but all apaches are running at this time. We sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support for assistance! -Patrick

Update Mar 1st, 5:30pm PST: The restore is still running smoothly on this server! As we previously mentioned, the load is still artificially high on the server due to the NFS mounted backups but all apaches are running at this time. We sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can look into this for you. – dan

Update Mar 1st, 10:00am PST: The restore is continuing to move along nicely! As noted previously, the load is still artificially high on the server due to the NFS mounted backups but all apaches are running at this time. Once again, we sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can look into this for you. – Justin

Update Mar 1st, 3:00am PST: The restore is continuing to move along nicely! As noted previously, the load is still artificially high on the server due to the NFS mounted backups, but all but 1 Apache and services are running. There may be slower than normal speeds until the restore completes. Once again, we sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can look into this for you. – Chalise

Update Feb 29th, 6:39pm PST: The restore is progressing well. The load is still artificially high on the server due to the NFS mounted backups, but all but 1 Apache and services are running. There may be slower than normal speeds until the restore completes. Again, we apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! – Seohee

Update Feb 29th, 3:04pm PST: The restore is progressing well. The load is still artificially high on the server due to the NFS mounted backups, but all Apaches and services are running. There may be slower than normal speeds until the restore completes. Again, we apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! – Seohee

Update Feb 29th, 7:30am PST: The restore is progressing, with an estimated 78% of the total data volume restored. The load is still artificially high on the server due to the nfs mounted backups, but all apaches are running and services running. There may be slower than normal speeds until the restore completes. Again, we apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! – Justin

Update Feb 29th, 4:00am PST: We are still seeing loads that are much higher than normal due to the slowness of the backup server mount. Our admins are continuing to look into this and, as previously noted, hope to have a solution in the near future, but currently don’t have an ETA on when this situation will be resolved. Again, we apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! – Chalise

Update Feb 28th, 8:20pm PST: The mount to the backup server is still slower than we would prefer, unfortunately, causing higher load and slowness. Our admins are looking into this and we hope to have a solution in the near future, but currently don’t have an ETA on when this situation will be resolved. We ran into problems with the load spiking that caused some temporary downtime. An admin has fixed this up and the restore is running smoothly once again. We apologize for the inconvenience. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! Thank you for your continued patience. – Seohee

Update Feb 28th, 9:30am PST: The mount to the backup server is still slower than we would perfer, unfortunately, causing higher load and slowness. Our admins are looking into this and we hope to have a solution in the near future, but currently don’t have an ETA on when this situation will be resolved. We apologize for the inconvenience. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! Thank you for your continued patience. – Justin

Update Feb 28th,3:45am PST: The data restore process is continuing to move along nicely, however, we are noticing that the load on the server is high which is causing sites to load slowly and also producing 500 errors/internal server errors. Our admins are working to try and stay on top of this, however, you may notice intermittent issues until the restore completes. We apologize for the inconvenience. If you find that you are missing data or see incorrect content on you site, please contact support so we can help! Thank you for your continued patience. – Chalise

Update Feb 27th, 7:23pm PST: The data restore process continues to run smoothly. However, if you find that you are missing data or see incorrect content on you site, please make sure to contact support so we can help! Thanks again for your patience! – Seohee

Update Feb 27th, 1:08pm PST: As previously noted, this server has been moved back to its old hardware. The data restore process continues to run smoothly, however, if you find that you are missing data or see incorrect content on you site, please make sure to contact support so we can help! Thanks again for your patience! – Brian S

Update Feb 27th, 8:17am PST: As previously noted, this server has been moved back to its old hardware. The data restore process continues to run smoothly, however, if you find that you are missing data or see incorrect content on you site, please make sure to contact support so we can help! Thanks again for your patience! – Chalise

Update Feb 27th, 12:50am PST: Our admin team was able to move everything back to the old hardware, and data is currently being restored from backups and is running smoothly. If you are missing data or see incorrect content on you site, please contact support with any details, so we can assist you with this further. Thanks for your patience while this restore completes! – Chalise

Update Feb 26th, 6pm PST: The restore continues as expected. All sites should be loading using backups data. If you see no or incorrect content on you site, please contact support with the specifics, so we can assist you. We appreciate your continued patience while we’re taking care of the restore. – Andrea

Update Feb 26th, 6:25am PST: Apache services have remained up and the machine is stable. If you notice missing or outdated data, or experience any other issues, please be sure to contact support! Thank you for your patience with this issue. – JJ G

Update Feb 25th, 8:45pm PST: The server is now back up and all apache instances are up. At this time all sites are running off mounted backups while the restore completes. You should not be seeing any 403 errors on your sites now. If you getting a 403 error, please contact our support team. The restore has been restarted  now and we will continue to update this post as it progresses. If you notice missing or outdated data, or experience any other issues, please be sure to contact support! – Oscar

Update Feb 25th, 8:00pm PST: We are not stopping the server and will update this post as soon as it’s back up. – Oscar

Update Feb 25th, 7:10pm PST: The restore process has been running for the last few hours. There are a few issues which are causing your sites from loading correctly off the mounted backups. We are going to stop the server briefly to resolve these issues. We will update this post when we are ready to stop the server and as soon as it’s back up. Thank you for your patience, further updates are forthcoming. -Oscar

Update Feb 25th, 5:30am PST: The restore process has been resumed and has been running without issue thus far. Websites on this server should be online at this time. Thank you for your patience, further updates are forthcoming. -JJ G

Update Feb 25th, 1:30am PST: We have been forced to stop this restore process for awhile, our admins are working to get it started once again. The data is still there and we will continue the restore as soon as we’re able.  We’ll be updating this post when that process is resumed. Thank you for your patience. -JJ G

Update Feb 24th, 6:00pm PST: This restore is continuing without any issues at this time. Sites should all be online at the moment. If you notice missing or outdated data, or experience any other issues, please be sure to contact support! -Dan J

Update Feb 24th, 6:05am PST: This restore is continuing without any issues at this time.  If you notice missing or outdated data, or experience any other issues, please be sure to contact support! -Patrick

Update Feb 23rd, 7:50 pm PST: Backups have been mounted to the new machine and the apache instances are now all online. Sites on this server should be up and running off the backups and the restore has begun. As mentioned before, backups are mounted via NFS so that your sites are up while the restore runs. Once your website is up, if you notice missing or outdated data, or experience any other issues, please be sure to contact support. We will continue to provide progress updates. –Seohee

Update Feb 23rd, 3:50 pm PST: Backups have been mounted to the new machine and the apache instances are now all online. Sites on this server should be up and running off the backups. As mentioned before backups are mounted via NFS so that your sites are up while the restore runs. Once your website is up, if you notice missing or outdated data, or experience any other issues, please be sure to contact support.  We will update this post again once all apache instances are up. –Dan J

Update Feb 23rd, 2:10 pm PST: Backups have been mounted to the new machine and the apache instances are starting up now. Once all apache instances are up customer sites will be up and running off the backups. As mentioned before backups are mounted via NFS so that your sites are up while the restore runs. Once your website is up, if you notice missing or outdated data, or experience any other issues, please be sure to contact support.  We will update this post again once all apache instances are up. – Oscar

This entry was posted on Thursday, February 23rd, 2012 at 12:28 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

62 Responses to “Shared web server ‘stampeders’ restoring from backups (RESOLVED)”

I’m on stampeders and all of my web sites are down, when I put in a support message I notice it says this next to my site addresses:

YOUR WEB SITE DOES NOT APPEAR IN THE APACHE CONFIGURATION FILE

YOUR SHELL USER WAS NOT FOUND IN THE WEB SERVER’S PASSWORD FILE

I’ve submitted a support ticket, please help!!!

I have same issue as Robert above:
YOUR WEB SITE DOES NOT APPEAR IN THE APACHE CONFIGURATION FILE

YOUR SHELL USER WAS NOT FOUND IN THE WEB SERVER’S PASSWORD FILE

Is there an ETA? Also, when was the last backup made? I did a lot of work on several sites last night straight into morning and would hate to lose all my work.

I did quite a bit of work on my site this a.m.
And, I just found out that work is gone.
I’m definitely not pleased with this situation.

@Robert, @Lori: This was due to the backups not being mounted yet. This was done by 2:10 pm this afternoon and your sites should be running again. If you have any issues, please contact support under Support / Contact Support in your panel.

@Julian: If you find you are missing any data, please submit a support ticket in your panel under Support / Contact Support and mention which sites and/or users so we can get that sorted for you.

@Dale: Your site is not on this server so the message above does not pertain to you. It looks like your site was compromised. I have moved your ticket over to our abuse/security team for further review. Also, if you are not sure which server you’re on, you can go to the Users / Manage Users page in your panel and check the ‘machine’ for your user.

xxx.com it’s not in my domain names but I have error Can’t add domain: already in our DNS system.

HELP

I’m now seeing this if I try browsing https://jaynaylor.com:
SSL connection error
Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don’t have.
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.

ssh to the related account fails to authenticate.

ps82699 – (DOWN AGAIN)
I’m tired.

2:10 pm PST : data mounted, apaches starting
3:32 pm PST : all my sites still down. Is it OK? Just asking.

A number of my sites are down. This is causing many angry customers and I need an answer ASAP as to what is going on and when it will be remedied. This is the message users get when they log onto an affected site:

Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”

@okashii: Your support message is being replied to right now, you should be getting an email very soon.

@McGroarty: I’ve gone ahead and restored your site manually. You’re up and running, though performance will be better when the restore process is complete.

@Ciro: I’ve replied to you via the support system, again my apologies for all the issues you’ve experienced.

@Ozh: You should start to see your site files restoring, it seems to be running for you right now.

can you bring the site i need back up? ugh!

While some of our client sites are up, we have 23 SITES THAT ARE DOWN!!!!

We are getting complaints from clients. Our own company website is down as well.

We need an answer now as to what is happening, why and what you plan to do about it. We need immediate restoration as well.

Error messages are either:

Site Temporarily Unavailable
We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”

OR

Forbidden
You don’t have permission to access / on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

These are sites getting daily traffic. This is not a cache issue as you tried to state initially.

Thank you!

We are on Stampeders and ALL of our files have been wiped. What’s going on here?

fwiw, it looks like a number of user directories are missing or inaccessible, ex. the users “bonk” and “metasvn” fail as below:

$ ssh bonk@bonkifiedart.com
bonk@bonkifiedart.com‘s password:
_ _
__| |_ __ _ _ __ _ __ ___ __| |___ _ _ ___
(_-< _/ _` | ' \| '_ \/ -_) _` / -_) '_(_-<
/__/\__\__,_|_|_|_| .__/\___\__,_\___|_| /__/
|_|
Welcome to stampeders.dreamhost.com

Any malicious and/or unauthorized activity is strictly forbidden.
All activity may be logged by DreamHost Web Hosting.

Last login: Sat Feb 25 10:06:37 2012 from 67.166.129.98
Could not chdir to home directory /home/bonk: Permission denied
-bash: /home/bonk/.bash_profile: Permission denied
bonk@stampeders:/$ pwd
/
bonk@stampeders:/$ cd
-bash: cd: /home/bonk: Permission denied
bonk@stampeders:/$

All of my sites are STILL down…I will happily be more patient if you can give us an ETA…I don’t understand why the restore is taking SO long. Even a temporary fix is better than an entire day down.

Same issues of all sites being down. Can’t login to FTP.

Whelp. Sites are STILL down. No updates on this blog. No response to support ticket or on Twitter. And of course one of my sites just got a huge blog mention. I know poop happens, but is releasing updated information really so hard?

Hello???? Anyone still work here? I don’t see how it can possibly take this long to restore a backup. I’ve got all of my sites down, all of my accounts locked out, and can’t do a thing. Ever hear of the word “communication”? Any update is better than this.

@Matt: The status was updated a few minutes ago. Can you let me know your domain name and I will be glad to check things out for you.

Y’all really need to message this more clearly.

It looks like the backup is from January 29th, but I haven’t seen that announced anywhere.

Also, after restoring some of my sites, I added the missing changes since January 29th from my own backup. Then this evening, that data overwritten with a backup from January 29th again. I’ve just finished re-reuploading the last month’s changes. Is that the last time this should happen, or do I need to keep watching?

Hi,

I’m on stampeders and my site is completely down, just a blank page, I’ve sent through a support ticket, anyone else having the same problem?

What a nightmare Dreamhost has become!!!

These sites are still down as of this morning:

http://cwrsystems.net/
http://professionalcountertops.net/
http://www.gioulis.com/
http://giospizzabistro.com/
http://devlinprints.com
http://www.kingofhearts.biz/
http://preservationofspace.net/
http://www.peaknutritionsolutions.com (completely down – at least it was partially back up yesterday afternoon)
http://consultforresults.com/ (big red warning that something is not right here)
http://professionalcountertops.net/
http://preservationofspace.net/

Please advise when you will have them back up! This is causing major problems for us and the clients!

McGroarty – I feel your pain. While some of our sites have been restored, many – some crucial sites – are still down. And the ones that have been restored, yes, it appears on my side as well that the updates go back to the end of January.

So we not only have angry customers and a poor communication from DH, but added work that may have to be repeated.

DH – as the others have stated – we need honest answers here as to what is happening. Your posts and support responses have not been accurate.

You tried to blame me initially, saying it was a “cache issue” on my side, when you knew exactly what was the cause.

Your status report on the top of this page says “all customer data shouldn’t be affected at all.”

I understand problems occur, but it’s how a company handles it when they do that counts. In fact, this status thread should have been moved to the top of the status list with these new problems, but I almost feel like you want to keep it on the down low so other users don’t see the seriousness of what is going on.

What’s past is past – but we are asking you to step up to the plate now and start communicating!!!

All wordpress and joomla based sites STILL are down. I’m VERY upset over the date of the backup because my main site just underwent a custom wordpress theme change and it’s all been wiped clean while I wait for the third party backup! What happened to the nightly backups??? Two open support tickets since yesterday and no response.

@Everybody with sites not working: we have had it confirmed with our systems engineering team that the restore is going as expected: all sites should be loading. Whatever you see on your site now is what we have on your backup server, so if it’s not accurate, please contact our support team through the DreamHost panel with the specifics, so we can assist you. All sites I was able to check through here are loading, and some of you didn’t post any identifying information, so we need that to check on your case.
@Karen – the status posts are listed in chronological order based on when the post was added, but remain ‘sticky’ as long as the case is open, that’s why it’s still listed on the main page. By no means are we trying to hide it, we need are customers to be aware of it in case they are affected.

Very disappointed by this outage. Once I made note, they got it back online, but it was/is reverted to an older version and all images are down. This is seriously not cool.

My site went down completely yesterday and I contacted support, they replied over 12 hours later with this. No apology about the outage, no mention of getting credit for these continued outages over the past few months and totally vague about whether or not these problems have been resolved. Images uploaded after Jan 21st were completely gone and when you went to my site it was just blank. Luckily a developer helped me fix the problem cause Dreamhost were of no help whatsoever. I’ve been with Dreamhost for over 5 years and I would say over the past year or more they’ve gone seriously downhill. Their customer service used to be top notch but now that’s gone downhill too. I’m currently exploring other hosts. I don’t trust them with my sites, email or data at this point. They seem like they’re getting worse by the second! I agree you with Karen I think this issue should be at the top of the page, Dreamhost’s communication is very poor and in the past they’ve also tried to blame me and my developers for cache issues. Once my site was down for many days and when my developer finally figured out the problem Dreamhost still denied it was their fault and actually accused me of spamming with multiple messages about my site being down. Deplorable customer service, just outrageous!

Yes, the missing data is most
likely due to the issue with the webserver:

http://www.dreamhoststatus.com/2012/02/23/shared-web-server-%E2%80%98stampeders%E2%80%99-restoring-from-backups-to-new-hardware/

As you know it was being restored from backups but if you still see files
missing at this point, you will need to upload your local backup as you
have already stated you are doing. After the issue with the server has
been resolved, we would be more then happy to do a cautionary scan of
your account for hacked site evidence. If you have any other questions
or concerns, please do not hesitate to contact us.

Thanks!
Jose G

@Robert: I was the support manager that replied to your most recent ticket response. I do apologize again for the premature and incomplete response from Jose. When dealing with downtime issues, the tech support managers handle any related incoming tickets, but it appears yours was answered prematurely by Jose. If you have any further questions or concerns for me, please reply to my email.

Carrying on from my Feb 23rd comment way up above in this thread, I must express my complete disappointment in Dreamhost. As a longtime client with many fully hosted domains, and domain registrations, I find it impossible to continue to recommend Dreamhost to anyone at this point. I will not make threats to pull my account and switch registrars and hosts, will not hurl expletives, but will challenge Dreamhost to provide their affected customers with no less than 5 months of FREE domain hosting for all of their affected accounts. Face it, it is a pain in the rear to switch registrars and likely by the time you do successfully initiate a move, your site would be back up anyway (albeit temporarily). Dreamhost, please fix this, and fix it now. I have been having issues steadily since October (check support), and this last one takes the cake. Once you/I get my site back to functional with all my missing images restored, and html updated to current, (which means ftp will be working again), I will write you privately with requests for free service until you agree to do so. If I have to drive to your offices myself, or use a real piece of paper and a stamp if email wont work.
Have a wonderful day, and please have my ftp ability restored by 4pm ET today so that my business doesn’t have to credit its customers for your inability to keep my companys web presence from doing its job.

my site is still down (over 8 hours now) and i have not heard any feedback from support. Can anyone assist?

My websites are still down (over 8 hours) and i have yet to hear from support. Can anyone from dreamhost assist?

Seriously? All my sites are down again? Someone asked me if it was a “lingering” problem… After five days I’d have to go with “festering”.

Over 12 hours and no response from support. Is anyone still on the clock? I try to empathized (working in a NOC for over 6 years), but this is getting kind of ridiculous! Come on guys!

I agree. I’ve been a dreamhoster for a long, long, time now, and the total lack of customer support at this point is beyond ridiculous. Stampeders has been down since the 23rd (at times it works, but not fully restored) and now, over 6 days later, all sites are down, files gone and no response from customer support in over 35 hours. I know its shared hosting and stuff like this happens, but enough is enough.

Ditto what has just been said. This situation is beyond belief. What are we supposed to say to our clients whose websites we’ve entrusted to DH: “Sorry, we picked an inept server company and now you are paying the price.” ?

We’ve had concerns over the years with some of the unprofessionalism and problems, but up until now, we have not been personally affected in a major degree as we are now. We should have taken those as red flags. But we like to be loyal and give people the benefit of the doubt because things happen, but come on already, we all know it’s the service after the problem that sets you apart.

This lack of customer service is threatening to destroy the business we’ve built up over the years and the confidence our clients have had in us.

Ever heard of RAID, DH? What about redundancy? Weekly and daily incremental backups? How do you have an OS failure that brings sites down for 6 days and when they do come up for a short teaser amount of time, your hand is revealed that the most recent backup you have (or used) is a month old? Why wasn’t there a redundant server to take over? Where are the failsafes? Why wasn’t the problem detected?

And even if you have excuses for all of this, what are you doing about the situation now? How are you getting sites back up (and staying up) in an expedient manner?

How could you possibly post “we hope to have a solution in the near future, but currently don’t have an ETA on when this situation will be resolved” and think that is an acceptable status report? Your lack of understanding of the seriousness of what this is causing your clients and our clients is deplorable.

Outages happen, but it would’ve been nice to receive some actual communication via email to at least advise users who are affected and provide some updates………rather than me noticing issues with my sites and having to chase it up even though it’s already being investigated.

@lachie: Unfortunately communicating via email for the number of customers on a shared webserver isn’t always practical, with the amount of time it can take to send out the emails (some providers throttle incoming, spam blocks, etc). We are looking into other methods of communication.

@Karen and others: Our apologies for the issues with this restore, it has not gone nearly as smoothly as previous ones. We do have hardware raids for all web and mysql servers, but even a raid is not a completely fault tolerant setup, and problems can occur. We have always tried to keep as up-to-date backups on separate servers so should any issues occur, we can both run the server from the backups temporarily and restore from them. In this case the backup servers raid was also not in the most ideal condition to do this, an issue we are doing a audit for and fixing up any issues we find.

At this time the restore is getting close to complete, apaches are running and we hope to have this completed and working normally on its own hardware shortly.

With multiple websites basically “down” because they are running outdated backups the restore is obviously extremely important to us. There is literally “weeks and weeks” worth of work hours we are poised to lose if it does not go successful and we are unable to upload changes to websites (just waiting) until the restore is done.

With the last update coming at 7:40AM I would imagine another post involving an ETA should be due.

The comment was “it would be on its own hardware shortly”.

We need honest answers. Please tell us realistically what to expect.

still at it DH, I now have what looks like an old version of my site, minus all one clicks, that is slower than half a snail. Do you have plans for the future to avoid this shambolic event recurring?

I realise you can’t give a certain answer, but using your professional experience, are we looking at hours, days weeks or never for this to be rectified.

May I suggest the $19 compensation you have awarded me is raised to something like a full years hosting free, and may I also suggest other users ask similar questions.

My web site is down AGAIN!!! Put in a support ticket, no response yet…

mine too, all gone again

I really feel like Dreamhost is having big big problems, I’ve been with them for many many years and while I have downtime now and again it’s only recently that the downtime has become more frequent coupled with no support response, they’re responses used to be quick and helpful. I don’t feel confident that my data is safe with Dreamhost and won’t all get lost. Right now my web site doesn’t load at all and my developer is trying to fix it but says he can’t even connect, that the load average is 578! I’m getting tons of customer complaints, I haven’t been able to accept orders in days so am losing a lot of money. Total joke. Basically I’m hoping they can get the site up and running (though this is doubtful) and as soon as they do I’m going to try to grab everything and put it on another server until they stabilize. This is really becoming a nightmare!

100s of my websites are down for last 7 days and they even didn’t reply to my emails, what a big mistake i did to host my websites at DreamHost. Never knew that a company can take 7 or more days to restore the server. 7 days are way too long, do you have a y ideaaaaaaaaaaaaaaaaaaaaaaa….

“Update Mar 1st, 3:00am PST: The restore is continuing to move along nicely! As noted previously, the load is still artificially high on the server due to the NFS mounted backups, but all but 1 Apache and services are running. There may be slower than normal speeds until the restore completes. Once again, we sincerely apologize for the inconvenience this is causing. If you find that you are missing data or see incorrect content on you site, please contact support so we can look into this for you. – Chalise”

THE RESTORE ISN’T SO NICE FOR ME! I’VE HAD NO WEB SITE AND NO RESPONSE FROM SUPPORT AFTER SUBMITTING A TICKET 5 HOURS AGO. IF YOU HAVE THE TIME TO POST UPDATES DO YOU NOT HAVE TIME TO RESPOND TO CUSTOMERS WHO ARE BEGGING YOU TO GET THEIR SITE BACK UP?! WE ARE LOSING CUSTOMERS AND MONEY HERE. OUR SITUATION IS DESPERATE AT THIS POINT ESPECIALLY CONSIDERING THAT WE’VE SUFFERED FROM OUTAGES OVER AND OVER AGAIN RECENTLY. WE’RE STARTING TO LOSE CUSTOMER CONFIDENCE AND WE CAN’T TAKE ANY ORDERS AT THE MOMENT BECAUSE WE HAVE NO WEB SITE. SO MUCH FOR BEING ABLE TO EAT THIS WEEK! IT’S NO USE TELLING CUSTOMERS TO CONTACT SUPPORT FOR HELP IF YOU DON’T WRITE THEM BACK!!!

Bad to worse. They just sent this because I mailed to say although my site is back it’s very slow. Can you believe this, I find it offensive. Please look at what my sites are, just some small images. If these sites clog their RAM / CPU they really are working from someones bedroom.

Under shared
hosting, we limit the RAM and CPU that your user can consume at any given
time. This is done to prevent any one user from dominating the server’s
resources and is there to protect you as well as limit you.

As I explained, you will need to use the tools provided here to try to
find the cause of this usage:

http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage

Under shared hosting, this is very normal and the only way to allow your
users more resources would be upgrading to a VPS or Dedicated Server.

caferoyalbooks.com
craigatkinson.co.uk

@Craig FYI, I’m in the UK myself and tried visiting your sites and they don’t load for me at all. And Dreamhosts’ response does read like they are trying to blame you for their problems…

I think this has been brought up before, where does Dreamhost host their site cause they never seem to go down while the rest of us suffer. How come their customers whether shared or VPS don’t get the same service???

It’s really terrible – amongst the worst service I’ve ever had for anything, and the level of blame is disgusting, coupled with an insulting $19 refund as a good will gesture. I’ve recommended so many people over 6 years who have signed up and made DH thousands.

Does anyone have any reasonable suggestions for hosts elsewhere, UK or US?

the dream has become a nitemare. do something before i loose all my clients…

I NEED HELP!!!!!!!! look what you say to me:
“Hi Daniel,

I apologize, I’ve gone through all of our backup servers, and have been
unable to find backups for any of these users. While we do our best to
maintain backups for all users, we can’t guarantee their availability,
and you’ll need to restore these from your own backups. I’m very sorry
that there’s nothing more we can do for you in this matter.

Thanks,
Brandon”

Hey guys, don´t say you don´t have backup, in the post about michaelkelly you said you had the info from the old hardware, well, there has to be the information, the sites that are online, is because i reuploaded them, I NEED THE INFORMATION TO BE AVAILABLE; I HAD SVN, I HAD MORE THAN 130GB THAT NEED TO BE RESTORED, THIS IS NOT MY FAULT!!!! I MOVED TO THE VPS TRUSTING YOU TO HAVE THE OLD HARDWARE BACKED UP, I NEED THESE USERS URGENTLY!!! I DON´T KNOW WHAT YOU HAVE TO DO, BUT I DON´T HAVE BACKUP OF THESE USERS, AND THEY ARE THE MOST IMPORTANT ONES!!!! PLEASE SEND THIS TO SUPERVISOR, OR MAKE MAGIC, I DON´T KNOW, IT´S SIMPLY IMPOSSIBLE YOU DON´T HAVE THE INFORMATION IF YOU HAVE THE OLD HARDWARE, I NEED THIS TO BE FIXED AS A MATTER OF DEATH, MY CUSTOMERS ARE KILLING ME!!! A LAWYER CALLED ME THIS MORNING, PLEASE HELP!!!!!!!!!!!!!

davellanedam
intermarket
parperf
queenmaster
totumo
chiky_fly

I just got his message after my last support ticket 13 hours ago! These status pages lead you to believe everything is SMOOTH! GREAT! JUST CONTACT US! but Dreamhost isn’t responding anymore…

Hello,

Your inquiry has been moved to the queue of a specific tech support team member (this is either because they are already familiar with your case or are the best equipped to assist you with this specific issue). They will respond to you as quickly as they can but depending on the complexity of the issue it may take longer than normal for them to get back to you (even in excess of 24 hours in some cases).

Thanks!
The Dreamhost Ticket Moving Robot!

Dreamhost,

At this point you are destroying my livelihood. I have not had a business web site in 3 days, I have made no money in this time, have lost customers and have had countless complaints from customers that I’m sure to lose. At this point people think we’re an unprofessional and untrustworthy business. Why can you not have the decency to respond rather than move my tickets to give yourself “more than 24 hours” to respond? I NEED MY WEB SITE BACK UP NOW I CAN’T WAIT MORE THAN 24 HOURS! At this point I think I’m due a year free hosting. Even that is not enough. You used to be a such a good company with great customer support, that’s all changed and judging from forums and responses to your status pages I’m one of many who feels this way. Why does your site never go down when these problems happen but all year I’ve had to put up with continued outages and no real solutions, explanations or apologies. This is beyond unacceptable, it’s horrendous business practice and I’m starting to wonder if you’re going down in flames and taking your customers’ web sites with you. You say in your status updates “all restores are going smoothly! Contact support if you have problems! Thanks! All is peachy folks!” yet when people do submit support tickets you don’t even respond anymore and when you do it’s unhelpful, vague and the problem isn’t fixed. How am I even to back up my site if I can’t access it?

Stop lying and messing around. When will we have our web sites again? What is REALLY going on? Do you have to have some special power as a customer to be taken seriously and be responded to? Do you have no respect or concern for you customers’ businesses?

I have been very patient and I don’t throw about insults but this is beyond the pale and as a business we’re extremely angry.

I REPEAT, WHEN WILL STAMPEDERS BE FULLY RESTORED? WHAT IS REALLY HAPPENING? WHY ARE YOU MOVING SUPPORT TICKETS TO BIDE TIME RATHER THAN BEING PROFESSIONAL AND RESPONDING? IF YOU’RE IMPLODING THEN WHY NOT BE HONEST AND SAY FOLKS, IT’S ACTUALLY NOT GOING SMOOTHLY, IT’S BEYOND OUR CONTROL AND THINGS ARE REALLY MESSED UP. WHY CAN’T YOU GET IN PEOPLE THAT KNOW WHAT THEY’RE DOING AND CAN FIX THIS.

Losing thousands of dollars in business is devastating, especially in these hard financial times. You are ruining businesses out there. Please let us know what is happening, please get my web site running, please get things under control.

I NEED A RESPONSE AND I NEED IT URGENTLY, BUT MOST IMPORTANTLY I NEED MY WEB SITE UP AND RUNNING NORMALLY BEFORE MY BUSINESS IS COMPLETELY DESTROYED.

If you’re on twitter they’re here @dreamhostcare @dreamhosttweets but they dont bother replying to that either

Robert, I have switched to a new host and you also need to do the same. Dreamhost is just being worse, can you believe, my projects are down for more than 1 week. I believe that 1000s of customers already left dreamhost and this company is going to shut down for sure.

@Robert: The restore is done. What is your website URL or webid and I can check your sites for you.

@Daniel: We resolved your tickets already on multiple posts.

@Zareh: Your sites are up and working without any issues.

@Sushil: Your site is still hosted here and it’s working fine:

Domain Name: RENSOFTGLOBAL.COM
Registrar: GODADDY.COM, LLC
Whois Server: whois.godaddy.com
Referral URL: http://registrar.godaddy.com
Name Server: NS1.DREAMHOST.COM
Name Server: NS2.DREAMHOST.COM
Name Server: NS3.DREAMHOST.COM

You guys are so full of it. You make it sound like everything was restored without a problem.

In reality everything is restored from Jan 30th (over a month old) and anything that was done during that time period is gone.

This is literally going to cost us thousands and thousands of dollars to redo. Unreal.

@Oscar I’d rather not state my site, though you should be able to tell what it is from my mail and support tickets. My site is still operating slowly and I just got an email from Jose with the below info. What I don’t understand is my site was behaving completely normally prior to this stampeders problem and since it’s happened I’ve lost data (which I’ve reloaded, like Jay all data entered after January had vanished) and I keep getting intermittent internal server errors. It seems very odd to me that my site was fine, this stampeders fiasco happened and now it’s very slow yet Jose is saying that it’s due to OUR memory usage? If this is the case why was everything absolutely fine prior to this and now it’s not? I’ve submitted a support ticket as well but it’s taking at least a day or so for our support tickets to be replied to so trying this route…

I’ve checked into why you’ve been receiving these internal server errors,
and it seems your scripts have been getting automatically killed by our
Process Watcher due to your sites going over Memory limits on the shared
server:

Sun Mar 4 19:51:17 2012 procwatch3 INFO: PID 18321 (php5.cgi)
volcanictongue:pg592305 – 15.17 sec cpu [idle php]: killed for uid ram
Sun Mar 4 19:51:17 2012 procwatch3 INFO: PID 23126 (php5.cgi)
volcanictongue:pg592305 – 0.18 sec cpu [idle php]: killed for uid ram
Sun Mar 4 19:52:38 2012 procwatch3 INFO: PID 23175 (php5.cgi)
volcanictongue:pg592305 – 1.94 sec cpu [idle php]: killed for uid ram
Sun Mar 4 19:52:58 2012 procwatch3 INFO: PID 23524 (php5.cgi)
volcanictongue:pg592305 – 1.02 sec cpu [idle php]: killed for uid ram
Sun Mar 4 19:53:18 2012 procwatch3 INFO: PID 21051 (php5.cgi)
volcanictongue:pg592305 – 7.42 sec cpu [idle php]: killed for uid ram

I would highly recommend that you follow the steps in the following wiki
article in order to reduce your usage:

http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage

Also, please be aware that just because this site is the one getting
killed, it isn’t necessarily the problematic one.

For instance, if Site A is using 90% of your allotted memory usage, and
then Site B attempts to use an additional 15%, Site B will be 5% over and
will get it’s script killed.

I’d also recommend looking into any 3rd party plugins you may be running,
especially if you happen to be running WordPress installs as they can be
notoriously poor at memory management. (Secret Tip: Removing them from
the plugins directory is more effective then simply disabling

If you have WordPress installs, 500 errors are often caused by overhead
in some tables, if you optimize your tables on a regular basis, this may
solve your issue as well:
http://www.webkeydesign.com/26/optimize-your-mysql-databases/

Lastly, you may want to look into DreamhostPS, as you will be able to
raise your memory limits to whatever limit you’d like.

And now my site is down yet again and I have no email access this time either! AAARRRGGHHH!

Ive lost connection to all my sites again ?????

Why isn’t my Joomla-based site connectcrew.com working? I didn’t change ONE thing about it before the backup and now it doesn’t work. No one is addressing this in my previous support tickets, so I’m bringing it here.

Hi Dreamhost,

Still haven’t received a reply from my support ticket from 2 days ago, what gives?

Well this morning the problem is a bunch of images are missing from my site. I will start reuploading them now but can you tell me why this is happening still? They were there yesterday. It appears to be images we uploaded this past Sunday and the Sunday before that are gone.

It would be nice to have at least one day where our site functions normally and images didn’t start disappearing and the site runs quickly and I still would appreciate someone getting back to us. We’ve been customers for a very long time but we can’t promise we will continue to be considering this level of service.

thanks

A couple of weeks ago my upload path was changed to include the term “restoring.home/xxxxxx” and along the way, there was lost content on my site. I was told that any missing data could not be recovered. So, I went through my site and uploaded the missing/outdated data/images.
Last night around 8:30pm ET, something “magical” happened. All of the files on my server reverted back to the newest being from Feb 23. So, the updates I made since then were gone.
And now, the “restoring” part of my upload path is now too, gone.
What a joke. One minute stuff is there, then gone, and Dreamhost saying that they apologize for missing data. I just cant imagine how bad this is for those of you who have websites that are very complex and require endless hours of work. Dreamhost has inconvenienced and let down so many customers.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host ‘homie-vserver131′ back up and running on original hardware ( RESOLVED )
8:50 am

Web VPS host ‘homie-vserver131′ back up and running on original hardware ( RESOLVED )

Posted (February 23rd, 2012 at 8:50 am PST) by Oscar

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver131′, we are moving the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver131’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 24th, 8:40am PST: Our Admin team has gotten the original machine back in working order. As the original hardware is now running without issue, guests have been started back up and don’t need to be moved to new hardware. All affected VPS guests should now be online, but please Contact Support if you’re still experiencing issues. -Brandon

Update Feb 24th, 5:47am PST: Due to high load on this VPS host server, we had to temporarily turn off all the VPS guests. We have brought them back online and the load on the host is doing much better! The restore is continuing now and we will continue to monitor the server for any trouble! Please Contact Support if you are experiencing any issues! -Patrick

Update Feb 24th, 2:35am PST: The backup drive has been mounted to the new server and all VPS guests are being created on the new server. Most guests have been created and are up and running, however, there are about a dozen guests that our admin team is still working on. The restore is currently running and most VPS guests should be up as well, however, there is a higher load on the host machine that may be causing slowness – one of our admins is working on that as well. We will provide further updates as we have them. Thank you for your patience! – Chalise

Update Feb 23rd, 8:22 pm PST: The backup drive has been mounted to the new server and all VPS guests are being created on the new server. Our admin team is still working to get the remaining few VPSes running. After the VPS guests are up, we will start the restore process. Please keep checking this post for further updates. – Seohee

Update Feb 23rd, 3:35 pm PST: The backup drive has been mounted to the new server and all VPS guests are being created on the new server. At the moment about 3/4th of the VPS guests have been created and our admin team are working to get the remaining few running as well. After the VPS guests are up, we will start the restore process. Please keep checking this post for further updates. – Dan J

Update Feb 23rd, 10:10 am PST: The backup drive has been mounted to the new server and all VPS guests are being created on the new server. Once the VPS’s have been created and started, your sites will be running off backups mounted directly via NFS. After VPS guests are up, we will start the restore process. Please keep checking this post for further updates. – Oscar

This entry was posted on Thursday, February 23rd, 2012 at 8:50 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

77 Responses to “Web VPS host ‘homie-vserver131′ back up and running on original hardware ( RESOLVED )”

Servers affected by this outage:

ps19862, ps19963, ps20044, ps20235, ps20240, ps20449, ps20468, ps20600,
ps20634, ps20702, ps20793, ps20958, ps21018, ps21443, ps21759, ps21956,
ps22656, ps22670, ps22940, ps22943, ps23141, ps23267, ps23306, ps26255,
ps27127, ps27469, ps29412, ps29991, ps30298, ps30380, ps30429, ps30570,
ps31145, ps33589, ps33611, ps33953, ps34086, ps34493, ps34939, ps35200,
ps37146, ps37428, ps38751, ps50729

my site is down because of issue? ps26255

@harineta: Yes, your sites are down due to this issue.

I don’t see my server on there, but it seems to be down as well: ps29921

you are weak server,, Can you work out how to compensate for lost damage?

@Steve: Your VPS is hosted on another host machine and is up and working.

We lost all the database settings for a style manager plugin for custom CSS DrMiller.com at around the time this server move happened this morning. I am trying to restore the site, but am getting 404 not found pages. I submitted a ticket to support for a callback about 10 minutes ago. Waiting.

February 23rd, 2012 at 10:13 am日本児代表 Says:

どうも、こんにちわ。しっかり管理せんかいな、あんたらまいどまいどダウンしとんなや。

ps 57609 by any chance?

sorry it was a plugin that corrupted the database. Thanks for Ryan at support for helping me track down the error.

Our PS in on Oscar’s list and can be be pinged on and off. However, we cannot access via SSH and our sites are down. Can you please post the exact steps that we should expect to rectify the situation and an approximate time frame?

Is it possible to buy a second VPS, and have it auto-failover? I for one would have no problem paying double to lessen the chances of a major outage like this happening…

How can i find out what server im using? All my sites are down too.

What a joke. What am I spending all this money on a private server for? It’s no better than shared hosting. 4 hours down already and counting. :(

@Benjamin Could you please submit a support request and include the details

@Lawson No, we do not offer this functionality at the moment.

@David You can login into our Panel and click the ” Account Status ” link. There is a field labeled ” Web Server ”

@Jon Sorry for the inconvenience this has caused. However, all vps guests should be online. If you are noticing downtime, please contact tech support

@brians: I’ve been down around five hours. It’s STILL down… not online. I entered a ticket five hours ago, and they just directed me to this thread.

In fact, when I tell the Web Panel to run a test, it says my server (ps29991) didn’t respond to the tests. It also says that none of my domain show up in the Apache configuration file. Why it Live Chat down? I really need to get this resolved.

I’m on ps33953- my sites are not online – can’t even ping. I’ve placed a support ticket… 3rd critical problem with my VPS sites in 2 months- totally unacceptable.

My site is still down, and all I’ve got are three canned responses from support pointing me at this thread which says it’s fixed when it’s not. I’m paying over $150 a month for web and MySQL VPS.

I first reported a problem at 8am PST and it’s now nearly 2pm. That’s almost six hours of downtime. Ducking fell I am ipssed fof!

What’s messed up is that I was able to pull a backup of my DB stuff, and I have all the files locally. I’m thinking about just changing hosting providers, considering the site has been down for 6 hours anyway…

I’m unclear, support tells me to read this thread, this thread tells me my site should be up. It’s not. Someone clarify this for me.

I wish I could give my website visitors an ETA for the site to be back up again, or just something…. anything. How long should it take support to respond to a critical email?

They haven’t posted an update in four hours and twenty minutes… I don’t think they’ve done anything.

Please, we deserve to know an ETA for this “episode”.
Thanks.

The status update after this one is the same issue for a different server. The status update for the one before this one is the same issue for a different server. That one has been ongoing for 15 hours, and is still not resolved.

Seriously, put us on a shared server until you get your act together… just get the site up.

God, I hope I’m not still here after 15 hours. I know they are probably doing all they can, but this sucks.

However, if it is going to take 15 hours, tell us now DreamHost, rather than in six hours time. We can then do other things rather than hanging on like lemons.

Send a tweet to the CEO at @DreamHostSimon. If enough people contact him, maybe he can light a fire under them and make them hurry it up (or at least give us an update instead of expecting us to sit here in ignorance.)

They just rebooted my VPS because it starting using too much memory… Which is interesting because it’s down.

SFTP is up. I’m taking this opportunity to download all my code.

I’ve received this answer:

Thank you for contacting DreamHost technical support. Unfortunately I can
not offer you an ETA at this time as the process takes time based on user
account and file sizes. Since we do not have an ETA and we are reliant
upon the machine to run through this process we must direct our customers
at this time to the status page. We are truly sorry for the delays and we
are working toward a common goal of returning your sites and services
back to live. But this is a time consuming process and we ask for your
patients in dealing with these issues.

Why they can’t just give us a loose estimate based on what’s happened before on similar sized servers? I’m in Europe and it’s past midnight now. What a waste of a fricking day.

I created a new VPS, changed the DNS for my domain, and just finished copying over the essential files. As soon as the DNS propagates my iOS app will at least be back up.

Twelve hours and counting…

Server memory is maxed, all files are missing, no update on ETA, no chat text support….

All files are missing, server memory is maxed, no chat text support… update please.

My site is back up, but ONLY because I made a new VPS and put my files on it… something Dreamhost should have done 12 hours ago. I’m pretty disappointed in the response, but I know that in the future I’m going to have to fix it myself.

>to Oscar Says

HEY, when u will finish back up ?

Total down time 14 hours and counting.

Just read the update from the other day regarding restoring VPS host server ‘homie-vserver207′ restoring from backups which was posted on February 18th, 2012 at 3:52 am PST

Update Feb 22nd, 2:56pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

If you do the math, the backup on that other server was completed FOUR DAYS later.

I’m not sure if we are in for the same ride, but I sincerely hope not.

Lawson Says:

I created a new VPS, changed the DNS for my domain, and just finished copying over the essential files. As soon as the DNS propagates my iOS app will at least be back up.

Obi-Wan you are very wise.

I wish DreamHost had the foresight to do something like this rather than their bull in a china shop approach.

My VPS isn’t listed in the list above and yet it’s been up and down for the past 8 hours. It’s been rebooted three times already.

Support has yet to respond to my ticket, going on close to seven hours now.

2012 is turning out to be an amazing year for Dreamhost customers.

Almost 16 hours now and still no ETA.

How can anybody make proper contingency plans if they don’t know how long an outage will last?

Well, DreamHost. I’ve simply had enough now, and will take my business elsewhere.

we should wait 4DAYS!???

I hope not, but just a deafening silence here.

http://www.host gator.com/vps-hosting/ anyone?

We are still not up…. any news on the last supposedly last few that are still down?

@TK: Sorry for the brief downtime there! You are back online now! See the incoming update above for details! Thanks!

My websites are still down.

The VPS is supposed to be up and running, but it is not.

24hs waiting.

For the record, my site is fully up and running on a new VPS. Luckily I had a separate backup of the files to jump-start the process.

This morning I moved my DNS hosting over to DynDNS.org. They have a TTL (time-to-live) of 60 seconds (and let me set it to whatever I want). Dreamhost has this fixed at 4 hours… so you’re looking at a maximum 4 hour outage if you have to switch IP addresses.

So, I’m still with Dreamhost, but now I have two VPSes (plus a third for mySQL). If the “active” app server goes down in the future, I can switch to the other one, and the DNS record will be updated everywhere in 60 seconds. Dreamhost REALLY SHOULD offer a service like this. I guarantee you people will pay more for that kind of recovery.

@patrickm

Just to be clear we are not up.

Nothing but timeouts.

ps21956

We are still not up. Constant timeouts.

Mine’s not fixed either. Over 24 hours now and no response from support to my emails. What the hell sort of service is this supposed to be. It’s a complete shambles.

Still no luck with ps50729, just FYI.

You said it’s all up, but ps29991 still isn’t up. I filed yet another ticket, but I’m glad I moved my crucial stuff over 15 hours ago. I’d like to get the rest of it, if you don’t mind bringing up the server.

Still not up :(

ps50729 is still not up as of 09:00 PST.

Please, do not lie to us anymore. ps27127 is not up.

Over a day now. Fecking joke. :(

server seems to be back up, but database connectivity (mysql server is listed as ‘puffer’) does not work- my php connections to db all fail.

Still nothing is working on my PS, and I’m getting so angry now. DreamHost won’t even respond to my emails to let me know what the heck is going on. What the hell kind of service is this? If anyone is reading this, please just this sort this out now. It’s making my website look like a joke, and it’s making you look even worse. Thank you.

Brendan and Patrick, you are asking us to contact support if we still have any issues, but they are not responding to our emails. ps35200 thank you.

Now back up. Hallelulah! However, to say I am p1ssed off with the last 24 hours is an understatement.

now working!?

This is pretty clear to me.
The last update says:
“Our Admin team has gotten the original machine back in working order. As the original hardware is now running without issue, guests have been started back up and don’t need to be moved to new hardware”

Conclusion: in 24 hours they weren’t able to move the data. Luckily, the original machine is back. But for how long?

It’s time to get another hosting company.

By the way… all websites are down. Again. The illusion lasted 5 minutes.

No wonder it took so long… The Dreamhost Team is painting a mural at a school. :-|

http://instagr.am/p/HZfSCBjTbp/

DO WORK GUYS

@MF @John Could you please indicated your VPS name ( ex. ps12345 )? That would help us troubleshoot your issue

@Luke @Lawson @AB @Diego Your VPS guests are back online. Please let us know if you notice any problems.

I’m trying to move the domain name from ps29991 (which is now back online) to ps86392 (which doesn’t currently work with SSH.) It fails, and I entered a ticket 5 hours and 9 minutes ago. Still haven’t been contacted about it.

6.5 hours since I submitted the ticket. Don’t say “Let us know” as if you really want to know, then ignore us once the tickets have been entered.

24 hours later, my Google ranking has bombed out of site, my traffic is 10% what it was before the crash, and God knows how much good will my website has lost. Plus, I’ve lost revenue due to no traffic over the past 24 hours, and of course less traffic over the next few weeks as I try and recover a respectable ranking in the search engines. No accountability from DreamHost, no communication, no recourse, no response to queries during the downtime, no ETA, no fecking nothing. Well and truly held by the bollards. Still seething to say the least. :(

17 hours and counting on my ticket about not being able to SSH in. Still no contact AT ALL from Dreamhost.

xxx.com it’s not in my domain names but I have error Can’t add domain: already in our DNS system.

HELp

By the way, SOMEONE at Dreamhost changed the admin password on our server. If you want to do anything from your old VPS like login to SSH or move it somewhere else, or FTP, you may not be able to. I’m now locked out of my account… This is ridiculous. Two and a half days after the outage started, and my stuff still isn’t set up right.

Thank goodness I moved over my critical stuff before they screwed the old VPS up again.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups (RESOLVED)
11:32 pm

Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 22nd, 2012 at 11:32 pm PST) by chalise

Due to issues with the RAID controller on the shared web server ‘michaelkelly’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared server ‘michaelkelly’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Feb 28th, 5:50pm PST: The restore is now complete. All data from the backups and from the old hardware has been copied over.  If you are missing any data, please contact support detailing what you are missing and we can restore it for you. Please be sure to Contact Support if you have any questions or concerns. – Oscar

Update Feb 28th, 9:45am PST: We believe the restore is complete, but want to have admins verify it just to be sure before resolving this issue as completed. All data from the backups and from the old hardware should be copied over, if not, please contact support detailing what you are missing and we can restore it for you. Please be sure to Contact Support if you have any questions or concerns. – Justin

Update Feb 28th, 3:40am PST: The restore process has just about finished, most users on this server should be fully restored now. As mentioned previously, we aren’t able to mount backups for this server, so if you are missing files you will need to re-upload them if you need them up before the restore process gets to your user. All data will be restored from our backups though soon, however. We’re sorry for the inconvenience this causes you and thank you again for your patience. Please be sure to Contact Support if you have any questions or concerns. – Chalise

Update Feb 27th, 8:34pm PST: We’re getting closer to the end! The restore process is still moving along. As we mentioned before, we aren’t able to mount backups for this server, so if you are missing files you will need to re-upload them if you need them up before the restore process gets to your user. At some point, all data will be restored from our backups though. We’re sorry for the inconvenience this causes you and thank you for your patience. Please Contact Support if you have any questions or concerns. – Seohee

Update Feb 27th, 5:40pm PST: The restore process is still moving along. As we mentioned before, we aren’t able to mount backups for this server, so if you are missing files you will need to re-upload them if you need them up before the restore process gets to your user. At some point, all data will be restored from our backups though. We’re sorry for the inconvenience this causes you and thank you for your patience. Please Contact Support if you have any questions or concerns. – Dan W

Update Feb 27th, 12:40pm PST: The restore continues as expected and is running smoothly. New data will not be overwritten, so you may update your sites as usual; you can also upload any missing data from your own local backups as well if you do not wish to wait for the restore to complete. Our apologies for the issues this has been causing. Please Contact Support if you have any questions or concerns. We appreciate your continued patience! – Andrea

Update Feb 27th, 8:20am PST: As mentioned in the previous updates, we’re not able to run your sites off backups during the restore process for this server, so you may notice missing content until the restore completes. Most data has been restored, and the process continues to run smoothly to restore any missing data. New data will not be overwritten, so you may update your sites as usual; you can also upload any missing data from your own local backups as well if you do not wish to wait for the restore to complete. Our apologies for the issues this has caused. Please Contact Support if you have any questions or concerns. We appreciate your continued patience! – Chalise

Update Feb 27th, 12:40am PST: Data on this server continues to be restored. Unfortunately, as mentioned in past updates, we’re not able to run your sites off backups during the restore process for this server, so you may notice missing content until it completes. Fortunately, most data has been restored, and the process continues to run smoothly to restore any missing data. New data will not be overwritten, so feel free to continue updating your sites as usual; you can also upload any missing data from your own local backups as well if you do not wish to wait for the restore to complete. Our apologies for the issues this has caused. If you have any issues or questions, please Contact Support. Thanks once again for your patience! – Chalise

Update Feb 26th, 12:20pm PST: We appreciate your continued patience while the data is being restored. As mentioned before, we’re not able to run your sites off backups during this process, that’s why you may see empty folders. Over 2/3rd of the data has now been restored, and the process has been stable for a while. The less than 1/3rd of you still affected on this machine should be also restored soon. In the meantime, you may upload new data, and it will *not* be overwritten during the restore process.

We apologize for all inconvenience this may be causing. Please Contact Support with any questions you may have! – Andrea

Update Feb 26th, 6:20am PST: The server has remained stable and the restore process continues to execute properly. If you notice missing or outdated data, or experience any other issues, please be sure to contact support! Thank you for your patience with this issue. – JJ G

Update Feb 25th, 6:30pm PST: The restore process is running without any issues. Please note: We did not mount backups  via NFS which is why you are seeing empty directories during this restore. As the restore progresses your users will be restored. Thank you for your patience. Please Contact Support with any questions you may have! -Oscar

Update Feb 25th, 6:25am PST: The restore process continues to execute smoothly. Please note that currently empty users will be restored in turn and are not currently mounted from backups. Thank you for your patience. Please Contact Support with any questions you may have! -JJ G

Update Feb 24th, 11:25pm PST: The restore process continues to execute. Again, the old data is not currently mounted but only being restored, so currently empty users will be restored in turn. Thank you for your patience. Please Contact Support with any questions you may have! -JJ G

Update Feb 24th, 5:50pm PST: The restore process is going as quick as possible. There are multiple rsync’s running to restore the files. That process unfortunately does not mount the old data, so the empty users will be restored in turn. Thank you for your patience. Please Contact Support with any questions you have! -Dan J

Update Feb 24th, 5:41am PST: The normal restore process is underway on this server. As some of you have noticed, there are users that are not mounted from backups at this time. We are running a separate restore process in conjunction with the normal one in order to get all of those users restored. That process does not mount the old data, unfortunately, so the empty users will be restored in turn. Please Contact Support with any questions you have! -Patrick

Update Feb 24th, 2:24am PST: The server is now mounted from backups and is restoring to the new hardware. The entire post above has been updated to reflect the change as well. All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help! – Chalise

Update Feb 23rd, 10:50 am PST: The file system check is still running on the old hardware. Until that is done we cannot start the restore process to the new server. We will continue to update this post as the fsck continues. – Oscar

Update Feb 23rd, 7:30 am PST: The backup drive which is mounted to the new server is having issues at this time. This is the reason some customers are seeing older data on their sites and some sites are not resolving at all. Due to this issue we are going to wait for the file system check running on the old hardware to complete. Once that is complete we will be moving all services back to the old hardware. We will continue to update this post as the fsck progresses. – Oscar

Update Feb 23rd, 6:05 am PST: This restoration is moving along and we are not showing any issues with the server at this time! If you are experiencing any issues at all, please Contact Support for assistance! -Patrick

This entry was posted on Wednesday, February 22nd, 2012 at 11:32 pm and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

112 Responses to “Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups (RESOLVED)”

The server that my website is on is down right now, I think. Site’s not loading.

@John – looks like that server was triangulum and that should be working properly now (our admin was working on it at the time). To check which server you’re on, you can go to the Users / Manage Users page in your panel and check the “machine” listed for your user. If you don’t see an open issue for that server, you can submit a support ticket under Support / Contact Support in your panel.

All of my sites are down. They are all on the MichaelKelly.

My site is on the michaelkelly server and this morning I came in to find my site had reverted to August 5, 2010. There were some unexpected xml files on the site as well. I’ve deleted them and am now reloading my entire site but am wondering why the timewarp happened. Thanks

@Pat: The last update I added to this post covers this briefly. We’re having some issues with the backup drive for this server. We’re waiting for the fsck on the old server to finish now so we can revert all services to it.

You know, it’s too bad there’s not a dreamhost version of Twitter’s Fail Whale. At least then anyone going to our sites would get a cute image and some sort of an explanation, and not gobbledygook or just plain nothing.

Trying for the constructive route, rather than the panic route as someone just posted a promotion of one of my sites, today. One of the sites that trapped in file system check-land.

All of my sites are down, and I’m not on michaelkelly. I entered a ticket, but no response yet… I guess it was time for my weekly outage. :-(

Same here – also not on michaelkelly but my sites are donw :(

I have problemsssss

Is criticaaaaaaaaaaaaal

please! I cannot loose information! pleaaassse!

I’d second that having the equivalent of a “fail wail” would be nice. Having content from 2010 on our homepage all morning is not good. I’d just remove the index page myself but I don’t want to jeopardize modifying the file, in fear that it would not be restored due to some sort of timestamp based merge. If the currently live stuff is just going to be blown away and restored by the most recent backup, please let me know.

@Dustin: Yes, we plan on restoring data from the old machine. The file system check is still running and we are waiting on that to finish.

Currently, 2:20 PM eastern standard time: My domains are still down from 5:30 AM. Please let me know the estimated time for resolution, ASAP…

@Gary: With fsck’s there really is no ETA as it takes a few hours to complete. We’re about half way done with the fsck at this time.

1. How long does this process?

2. My data will be lost?

Please help me …
I have a severe problems with this process. My clients will initiate a lawsuit

Thanks ………….

@Oscar: A rough estimate based on the post time (11:30pm) and your comment at 11:02am (12 hours later) about the fsck being half-finished would be that it takes about 24 hours before you can start the failover to the new hardware? About how long will it take to complete the failover once the fsck is done?

@Peter The filesystem check is still moving along. Unfortunately, we cannot provide an ETA at this time. We really appreciate your continued patience.

1. How long does this process?

2. My data will be lost?

I’m so hurt !

How about an update!?!?!?! This is quite frustrating!!!

How long does this process? we are desperate

I’m sorry, but the backups should have been mounted to different hardware hours ago. This is beyond ridiculous. Why no failover? Even if it isn’t instant, 1 hour of downtime beats 24 hours always.

You lost a customer over this, not that you’d care. I’ll never trust you guys again for anything critical.

My website is still or again down. Very disappointing. I am with Dreamhost since about 1998, but these last few months have been quite bad and this almost 24h outage of my website is a disaster!

My svn server is still down. Please look into it.

Ladies and gents;

I have not been about to access the admins for the past 24 hours… Is this matter solvable? Please advise me what I can expect going forward… Thank you

I have not been able to access my admins for the past 24 hours… Is this matter solvable? Please advise me what I can expect going forward… Thank you

An update would be wonderful – I have a couple of very unhappy clients wanting to know.

Some news about this?

Before I submit an irate Support Request, exactly how much longer is it going to take for this to be fixed–only 1 of my 16 sites is appearing when I ftp into my account from the webpanel and even that one shows only the favicon files–no wordpress, no content, no nothing.

@iPhonedoktor.ch, @Barinder, @Gary James, @Katy, @Eder Acevedo, @Jenn —
Please see our most recent update to the post above for more details on the situation. If you have any specific concerns or questions, please contact us through your Support > Contact Support page so we can help you directly with your issues! Thanks!

But … I still I have sites like: http://www.centrodeidiomasamericano.com

These will be restored today?

Thanks.

But .. I still I have sites like: http://www.centrodeidiomasamericano.com/

These will be restored today?

Thanks

Hello,

Today, in this page: http://www.dreamhoststatus.com/2012/02/22/shared-web-server-michaelkelly-restoring-from-backups-to-new-hardware/comment-page-1/#comment-268506

Chalise says:
“All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help!”

I have sites down yet, with the message “Site Temporarily Unavailable” as:
http://www.caequinos.edu.co/
http://www.centrodeidiomasamericano.com/
http://www.colegiodivinosalvadormedellin.edu.co/
http://www.marketingintegral.com.co/
http://www.eserna.com/

and more….

This sites will be restored soon?

I have sites with old información !!! as:
http://www.activoscosmeticos.com/
http://www.moviesyvideos.com/

and more …

This sites will be restored soon?

I hace without información as:
http://www.santa-ana.co/
http://www.elgrifo.co/

This sites will be restored soon?

please answer me.

This situation is critical

This process has created many problems for me and this companies.

Thanks.

Eder Acevedo

Hello,
Today, in this page: http://www.dreamhoststatus.com/2012/02/22/shared-web-server-michaelkelly-restoring-from-backups-to-new-hardware/comment-page-1/#comment-268506 Chalise says:”All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help!”

I have sites down yet, with the message “Site Temporarily Unavailable” as:
http://www.caequinos.edu.co/
http://www.centrodeidiomasamericano.com/http://www.colegiodivinosalvadormedellin.edu.co/
http://www.marketingintegral.com.co/http://www.eserna.com/

and more….

This sites will be restored soon?

I have sites with old información !!! as:
http://www.activoscosmeticos.com/

http://www.moviesyvideos.com/
and more …
This sites will be restored soon?

I have sites without información as:
http://www.santa-ana.co/
http://www.elgrifo.co/

This sites will be restored soon?please answer me.

This situation is critical

This process has created many problems for me and this companies.

Thanks.
Eder Acevedo

This is seriously ridiculous!!! Trying to wait patiently but I have clients who are getting very upset.

Today, in this page: http://www.dreamhoststatus.com/2012/02/22/shared-web-server-michaelkelly-restoring-from-backups-to-new-hardware/comment-page-1/#comment-268506

Chalise says:
“All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help!”

I have sites down yet, with the message “Site Temporarily Unavailable” as:
http://www.caequinos.edu.co/
http://www.centrodeidiomasamericano.com/
http://www.colegiodivinosalvadormedellin.edu.co/
http://www.marketingintegral.com.co/
http://www.eserna.com/

and more….

This sites will be restored soon?

I have sites with old información !!! as:
http://www.activoscosmeticos.com/
http://www.moviesyvideos.com/

and more …

This sites will be restored soon?

I hace without información as:
http://www.santa-ana.co/
http://www.elgrifo.co/

This sites will be restored soon?

please answer me.

This situation is critical

This process has created many problems for me and this companies.

patrickm Says: “If you have any specific concerns or questions, please contact us through your Support > Contact Support page so we can help you directly with your issues!”

There seems to be an issue with Support not responding to our direct concerns, questions and requests sent through the Contact Support page. It has been over 24 hours and I have received nothing more than a robot response to one of my requests. I also have not received my requests callback for an update on the issue.

My clients are losing thousands of dollars in advertising over this issue. A direct response from Dreamhost to my specific questions is the least I expect.

Today, in this page: http://www.dreamhoststatus.com/2012/02/22/shared-web-server-michaelkelly-restoring-from-backups-to-new-hardware/comment-page-1/#comment-268506

Chalise says:
“All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help!”

I have sites down yet, with the message “Site Temporarily Unavailable” as:
http://www.caequinos.edu.co/
http://www.centrodeidiomasamericano.com/
http://www.colegiodivinosalvadormedellin.edu.co/
http://www.marketingintegral.com.co/
http://www.eserna.com/

and more….

This sites will be restored soon?

I have sites with old información !!! as:
http://www.activoscosmeticos.com/
http://www.moviesyvideos.com/

and more …

This sites will be restored soon?

I hace without información as:
http://www.santa-ana.co/
http://www.elgrifo.co/

This sites will be restored soon?

please answer me.

This situation is critical

This process has created many problems for me and this companies.

Thanks.

The Contact Support is ignoring my tickets … you aren’t answer me !

No sign of any of my sites working yet…

all site are empty ….. Esto es muy frustrante !

all sites are empty …esto es frutrante!!

Today, in this page: http://www.dreamhoststatus.com/2012/02/22/shared-web-server-michaelkelly-restoring-from-backups-to-new-hardware/comment-page-1/#comment-268506

Chalise says:
“All sites on this server should be up and running. If you notice old or missing data, please Contact Support so we can help!”

I have sites down yet, with the message “Site Temporarily Unavailable” as:
http://www.caequinos.edu.co/
http://www.centrodeidiomasamericano.com/
http://www.colegiodivinosalvadormedellin.edu.co/
http://www.marketingintegral.com.co/
http://www.eserna.com/

and more….

This sites will be restored soon?

I have sites with old información !!! as:
http://www.activoscosmeticos.com/
http://www.moviesyvideos.com/

and more …

This sites will be restored soon?

I hace without información as:
http://www.santa-ana.co/
http://www.elgrifo.co/

This sites will be restored soon?

please answer me.

This situation is critical

This process has created many problems for me and this companies.

Thanks.

esto es realmente desesperante, en serio no parecen una empresa seria.

I know you get what you pay for, but Dreamhost is by far the worst hosting and worst service I’ve ever dealt with. It’s been 36 hours and there has been no response to my support requests or any meaningful response here. I wouldn’t even choose to host a low-traffic or personal site on Dreamhost let alone any sort of paying client — which I luckily moved off of Dreamhost years ago.

DH asked me to reduce my data storage on michaelkelly recently. I guess just letting them lose it all is good enough. :)

All you people that say you’re losing thousands of dollars and/or have upset clients should consider investing a little more into your hosting service. Also, it’s not like DNS is broken. Upload your so-awesome website to another server and point your DNS somewhere else. Instant fix!

I’ve been with DH since 2003 and I’m not going anywhere. At least they make an effort to fix things when they break, even if they don’t have failover capabilities. You gotta pay money for that, not some $7/month.

…just wanted to say hello to my fellow shared hosters :) i’m curious how many times this page has been refreshed since this mess started.

@DH – my fiancé wants to know if you’ve tried turning it off and back on again? ;)

To the other Jen–countless times, I’m sure.

While I’m no longer getting error messages on my sites (so the server is somewhat up, now), I’ve yet to see a shred of data restored. And reading that last update with it’s talk of empty users and not mounting the old data has me wondering if we’ll ever get it back the way it was.

Michaelkelly is going down for reboot now (Fri Feb 24 10:24:22 2012)

This may mean we’ll be back up in a few minutes. I found all my data in /home.restoring/(username) and it looked as recent as last week, so I think they’ll just fix the mount points and we’ll be good to go here in a few.

Server back online, logged in.

Doesn’t look happy yet. The mount point for /home has 4.7T free, while when we were up and running it only had about 400G free. No one has logged back in yet it seems. Looks like they might be symlinking the home directories to an NFS mount while they restore the data but I don’t see that process starting yet.

fingers crossed.

happyyyy??? my sites are down yet!

The restoration process appears to be slowly rolling along. This can be viewed from a shell by using the ‘df -h’ command. You will notice that the mount point for /home is slowly increasing in used size. It appears to be increasing by about one gig every minute and a half. Right now it’s at 709G. It’s difficult to estimate when it will be finished and what order it is copying in. It may be alphabetical by username but more likely is age of folders.

My best recollection suggests that there should be about 4TB in use on this server, however the NFS mount they’re restoring from suggests a 1.4TB capacity. Maybe I’m overestimating data usage or they have multiple backup volumes to go to. If they really have 3TB of data left to copy, at this rate we’re looking at another 3-4 days before all the data is restored. I hope that’s not the case.

The good news is that your site will be up the moment it is copied into place.

The data they’re restoring appears to be up to date, or less than a week old. About two weeks ago I deleted a bunch of folders and I can see they’re not in the restoration folder.

For folks whose sites are not back up yet and you need your data, FTP in and navigate to /home.restoring/username and you’ll see your stuff. I tried to symlink sites within my home folder to their restoration location but I don’t have permission.

@DH, for users you haven’t restored yet, a little “ln -sf /home.restoring/username /home/username” would be appreciated. Thanks.

I cannot see my data on home.restoring.

I Cannot see my data on home.restoring
some problem?

Ray, you rock! I’d say you deserve the title of honorary dhsupport but, well, that might not come off as the compliment it’s intended as ;-)

I’m just glad to see updates and hints that something is going on!

[michaelkelly]$ ls /home.restoring/’user’
ls: cannot access /home.restoring/’user': No such file or directory
user is my name … my data isn’t here … why?
thanks.

Anyone know the actual status of things… I have client sites down and they are anxious to get them back up.

That may go with the theory that their backups are spread across multiple volumes and yours isn’t mounted yet. I can cd to /home.restoring/myusername right now (you didn’t actually use the ” marks right?) and all my stuffs are there.

I just checked, and it doesn’t work with FTP because there’s a chroot jail, you can’t get up past your user dir. HOWEVER, if you have ssh enabled for your user (shell instead of ftp only) you can use SFTP. FileZilla is a good client for this.

@Eder: If your username starts with a p (seen from running “last -a | head”) then it appears your folder is in /home.restoring/ but I obviously can’t see anything in it. It does seem to exist though. Try running cd /home.restoring/p***** (replace ** with the rest of your username) then do an ls.

ey I know about commands ;)
believe me, many of my sites are not there.

for example … http://www.santa-ana.co
[michaelkelly]$ cd /home/santaanaco
[michaelkelly]$ ls
santa-ana.co
[michaelkelly]$ cd santa-ana.co/
[michaelkelly]$ ls
favicon.gif favicon.ico

but in your restoring:
[michaelkelly]$ cd /home.restoring/santaanaco
-bash: cd: /home.restoring/santaanaco: No such file or directory
[michaelkelly]$ ls /home.restoring/santaanaco
ls: cannot access /home.restoring/santaanaco: No such file or directory

where is?????

some sites are in home.restonring, many others are not.

?????

@Eder just making sure :)

I think they’ve got 4TB of data to restore spread among three to four NFS mount points, and they’re only on number one, which is why some data is there and some isn’t. It looks like they may be about halfway through the first volume though, but that’s still 17 hours to go before they switch to the next one.

Maybe they shouldn’t have put 2500 happy dreamhost users on one server without a reasonable recovery plan. But, who am I to judge.

whaaaaaat?
Want to kill me from a heart attack?

Just tell me something ….
My Sites can be recovered with the updated information?

Por Dios! estoy que muero de la angustia.

@ALL Your data is in the process of being restored. Some users sites are already back online. Since we had available backups and was able to create a NFS mount to the backup machine.

Due to the age of the backup, some users data did not exist on the backup machine. Have no fear! The old hardware was able to be saved. We are in the process of copying data from the old hardware. Please understand that copying TBs of data is going to take some time. Thanks in advance for your continued patience.

omg this is becoming unbearable… the one thing i don’t have on back up is the thing my potential client is waiting to review. ugh. tick tock tick tock.
Some reassurance that newer files will be fully restored would be well received.

ETA on getting this fixed!?!?!

My website was totally down earlier, it’s back up but all my jquery stuff is broken. Plus I’ve noticed at least a couple files totally missing on my FTP site.

Is there any official update on the status of the restore? When we can expect sites to be fully back online? Is the process 50% done, 5% done, 90% done? Last official update was several hours ago.

Let us know by when everything will be normal,many of our sites are just showing Index with two files favicon,where is the website content ?
Let us know by when we can see our websites back.

this is the longest outage (2 days) I ever had with dreamhost in last 4 years.
I hope you guys will get everything up and normal.

From my outsider’s perspective, the home folder has grown to 823GB at 5:28pm PST, and it was at 709GB at 10:50am. So, that’s 20 gigs an hour.

I THINK the home folder is supposed to be 4.8 Terabytes, judging by /dev/sdb1 being 5.3TB in size and two weeks ago it being at 91% full. That is, unless this new hardware has a much larger capacity than the old hardware.

4TB at 20G/h is 200 hours, or roughly 8 days. Not nearly fast enough.

An alternate theory is that the user’s data is actually only about 1.4TB, which is the used capacity of the NFS share they’re currently copying the data from. If that is the case, it means there is about 29 hours left, or sometime tomorrow night. Better. But not good.

Either way, we’ve got some time to wait… Can someone from DH confirm how much data there is to copy?

8 days??? this is imposible …

Can this be copied faster?

Can you mount two NFS at the same time?

We can not wait that long ….

Please understand that is important and critical! ………..

xxx.com it’s not in my domain names but I have error Can’t add domain: already in our DNS system.

HELP

@Jen: It looks like the content under the domain linked to this comment has been restored. I sent you an email to follow up.

@Shawn: Unfortunately we don’t have an ETA on when exactly this will be done, since the amount of time it will take also depends on the size of the files being transferred. A large amount of small files, commonly used for websites, will take longer then just a couple very large files.

@Scotty: I made sure your data from the original hardware was synced. All your sites seem to be responding properly.

@Jason: As mentioned, we can’t really generate an ETA as it depends on more then just the amount of the data being transferred. I’ve gone ahead and made sure your sites are responding as well. If you continue having problems please let us know.

@Ray: I checked with our admins, we’ve got 3.2TB of data to transfer in total.

@Eder: We are copying data over as quickly as possible right now. Please contact support with any questions regarding problems with your services.

@okashii: Please contact support, this post is for the michaelkelly shared web server restore. You can find the support section here: https://panel.dreamhost.com/index.cgi?tree=support.msg&

@jj still have outdated content, username is dwwebmaster, if you can sync the data from the original hardware it would be much appreciated. Thanks.

The only one I need really, really urgently is servingsuggestion.co.uk – it’s showing the directory of but no files other than the favicon… I definitely can’t wait 8 days for this one!!

@jj “contact support” isnt answer our questions.
They are ignoring us.

Sont say please again that we should contact support, is stupid!.

I’ve been with Dreamhost for over 8 years, I trust them as my host, so much so I host all of my domains with you…..what a bad decision that was, because for the past two and half days every single one of my sites has been down and now it looks like they will be down for another 3-4 days.

What a complete disaster. As soon as my sites are back I will be moving very swiftly to another provider, this has been a total farce.

What is your “best guess” (hours / days) before this is fixed /restored. I need to make new arrangements for some of my sites ASAP.

As with Grace above, I’ve been with DH for over 10+ years, most of my sites are hosted with DH. I’ve been very happy webmaster with DH services,… up to this year, as a resul of the two major gliches (FTP password hack and server infrastructure issues.)

I’m sticking with you guys but I’d like some expectation or guestimate on timeline asap.

I.m sticking with you too …

is obvious that we are bad webmasters because we havent a contingency plan for these problems.

But, this should not happen and is in very poor taste by DH ….

DH has been thinking about some compensation for these damages?

Now I have some site running smoothly, thanks. But I have many site down yet. I’m dying!

Eder ! ederaam@linuxpreview.org

I.m sticking with you too …

is obvious that we are bad webmasters because we havent a contingency plan for these problems.

But, this should not happen and is in very poor taste by DH ….

DH has been thinking about some compensation for these damages?

Now I have some site running smoothly, thanks. But I have many site down yet. I’m dying!

Eder – ederaam at linuxpreview dot org

Literally none of mine are back up yet. Have had to transfer one client away and give her compensation. Hope DH will cover that!

@ALL: For those asking on an ETA, the best information I currently have tells us that it could take 2 more days. Again, this is a rough estimate based on the total amount of data involved.

@Jim: Please see the above.

@Dustin: That one just went through the restore process and you should be set. Let us know if you still have trouble by sending in a support ticket.

@Grace: Your data is being restored now and you should be back completely very soon.

@Eder: I checked your account and your machine is not at all linked to the problem machine in this post. Please contact support with the details of your downtime and we’ll be more then happy to help you out.

https://panel.dreamhost.com/index.cgi?tree=support.msg&

@Katy: I located your account and started the transfer of your files as well. You should be up again quite soon, depending on the total amount of data involved.

Can you help me with redfacultadescontaduria.org please? thanks !!

Dreamhost staff: I have been a loyal customer for 7 years and host over 70 sites with you. They are all down. I can’t even ftp my data back to them because your restore is locking them out. This is absolutely terrible planning. I just rolled out new products to hundreds of clients who are now getting errors or the index of 2 files on the new site; critcle sites for me are fatburnerup.com and duncantooleyhypnosis.com
You have got to do better than 2 more days!

Were you just kidding with this statement from the original message? “During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.”

If there is no data or you get “httpd error”, this is not “ON LINE” in anybody’s vocabulary.

thank you thank you thank you @DH. i am out of limbo <3

JJ
Can you please get my files transferred quickly! I am desperate to get some up for an event launching this weekend!

I love you guys and have no intention of leaving. I know that all shared hosts have problems from time to time. I do have to ask, however, is there any hope of any type of compensation for this extended outage?

@ Duncan, I’m .copying your files now, I apologize for the problems. In the future, shoot us a support message, we’ll get to it much faster for ya!

@ Ryan, I don’t see any of your account info, can you let me know what domain name you have with us?

@ Brandon, we’ll be happy to offer a credit, please submit a support request and we’ll handle it for ya.

jj – ryanwaller.com is one – do you need all? i’m #38263 “rwaller”

It’s 7:00pm Saturday Night (EST) – I’ve been down since Wednesday evening.
I need an alternative asap… are we trying to boil the ocean??

Hi guys, i´ve been a happy dreamhost customer since 2006, because of the problem i upgraded to VPS for files and mysql yesterday and i still don´t get my backup restored, can you please help me out? Also, because of the outage i would really ike you to give me some coupon, or credit, or something, many of my customers have been loosing money with this problem, and many have cancelled their hosting with me, this has caused a lot of trouble, so it would be nice from you to have some bonus for you loyal customers like me. PLEASE HELP ME ON RESTORING MY USERS FROM MICHAELKELLY TO ps86583

@Daniel: Unfortunately you will have to wait for the restore to finish before we can restore your files to the VPS. Once the restore is done, we can then rsync the files from the old server to your new VPS.

You keep saying “Contact Support”–well, support isn’t replying! I’ve never had a problem getting a response in a timely fashion before, but my Support Request from Friday morning is still unanswered. Not even a courtesy ‘we’ve received your request and will get back to you as soon as possible'; just nothing! I think that, combined with these vague updates, is the most frustrating part of this whole debacle.

JJ, Brian, Oscar::

Please help me with this transfer. My account info is above. I need this resolved ASAP!

Ryan

I have new domains/websites to create and work on today. My account currently points to MichaelKelly. How can I point my new stuff to a new web server.

“Contact Support” isnt answer my questions :/

@Ryan: Your files are currently transferring, you should be back soon.

@Jenn: Your files have now been restored and your sites are responding. I replied to you through email as well. Please let us know of further issues!

Thank JJ – But what might soon be? It’s been soon for 4 days now.
If I place any files up now, will they be overwritten or slow the process?

@Ryan – I followed up on your support request.

@andrea I saw but nothing has been resolved.

2.3TB restored and counting out of 3.2TB. That’s about 45 hours more to go until everyone is back online by my estimate.

What’s the opportunity cost of having all of one’s sites down for 6 days? Surely it’s at least worth a couple of free years of hosting, or maybe an unlimited space backups user? I would go for that.

Josh should come to my house with an 18 year old scotch and apologize in person for overselling and poor planning.

@Ryan – we received your email with the requested details. It’s being addressed now, but please note, it may take a little while to manually transfer your data, depending on its size. We’ll follow up again in email.

@Ray – we have an uptime guarantee with the details on compensation for outages on http://dreamhost.com/web-hosting/ under “Satisfaction Guaranteed”. Unfortunately, though, no 18-year-old scotch.

My sites are all back up and running this morning. Thanks, guys! W00t! Hopefully, you’ve figured out what the cause was and can ensure it doesn’t happen again.

2.9TB out of 3.2TB. That’s 91% of the way there! My sites are back up. For anyone still waiting, your sites could come up within the next 15 hours.

So in order to do the 100% uptime guarantee, you have to have opened a support case 6 days ago? Really? I did not open a support case as I saw this page and knew that it would do no good to flood you with more support cases for an issue you already knew about. Is this really how it is going to work?

-Kirk

@Kirk – I agree. I think that was written before they came up with dreamhoststatus.com. So, I think they should honor a time reimbursement from the date of this article, as me and any user would check this website, see the article, and decide not to open a request because it’s already been addressed.

So from this posting until my sites were up was a total of 4 days and 11 hours, therefore 107 days of hosting credit. Would that be acceptable @chalise? Thanks.

@Kirk and Ray – The uptime guarantee limits to up to 10% of your next pre-paid hosting renewal fee, so I’m afraid the 107 days wouldn’t apply. I did add the maximum amount of credit to Kirk’s account regardless the lack of actual ticket. Ray, I don’t have your info, but if you send in a support request referring to this comment, I’d be happy to credit your account as well.

Thank you. So what do I do if only part of the data on my account has been restored? My site are all back up, but other directories that were in my primary user’s home directory are still missing. Shall I open a support request on that and if so, how will it not get lost in the shuffle of everything else with this server or should I just be patient and see if the rest of it shows up at some point in the near future?

Thank you !!!!!

@Kirk – Open a support ticket and I can restore the data from the old server for you.

Please help!!! i´m in real trouble! i need you to restore some of my users, my support ticket is: #4781223

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware (RESOLVED)
5:09 pm

Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware (RESOLVED)

Posted (February 22nd, 2012 at 5:09 pm PST) by Dan J

Due to issues with the RAID controller on the web VPS host server ‘homie-vserver89′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver89′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 8:52pm PST: Our admin team have decided that the server appears to be healthy now and this was just a false alarm. We do not need to move homie-vserver89 to new hardware at this time. -Dan J

Update Feb 22nd, 6:08pm PST: Our admin team were able to get the old hardware online again and are monitoring the server it to determine if they need to move the server to new hardware. All VPS are currently online. We’ll have more info in a little while. -Dan J

 

This entry was posted on Wednesday, February 22nd, 2012 at 5:09 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware (RESOLVED)”

ps14497 ps14514 ps14539 ps14574 ps14608 ps14615 ps14645 ps14660 ps14694 ps14703 ps14706 ps14707 ps14712 ps14871 ps14872 ps14884 ps14929 ps14931 ps14932 ps15088 ps15202 ps15207 ps15293 ps15304 ps15642 ps17018 ps17080 ps18332 ps18335 ps18388 ps25583 ps26790 ps27670 ps27778 ps28144 ps30016 ps30421 ps30798 ps33963 ps41362 ps43803 ps54721 ps55793 ps56467 ps64525 ps72227 ps82336 ps83884 ps84664 ps85806

I hope it will not take days.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled maintenance for Ajaxplorer (WebFTP) tonight @ 8pm PST (RESOLVED)
3:39 pm

Scheduled maintenance for Ajaxplorer (WebFTP) tonight @ 8pm PST (RESOLVED)

Posted (February 22nd, 2012 at 3:39 pm PST) by Seohee

We will be upgrading Ajaxplorer (WebFTP) tonight, which is estimated to take about an hour. This is regarding the WebFTP link found at ‘Manage Domains’ in the DreamHost control panel. During this time, it is possible that WebFTP will not be functional. You may want to use a different (non-web-based) FTP client during that time. This wiki article will give you further details on other available FTP client software.

We appreciate your patience and are sorry for the inconvenience. We’ll make sure to update this post once the upgrade begins.

Update Feb 22nd, 8:29pm PST: The Ajaxplorer update has completed. Everything appears to be working. If you find any problems then please contact support. -Dan J

This entry was posted on Wednesday, February 22nd, 2012 at 3:39 pm and is filed under DreamHost Features, FTP Outages, System Changes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Scheduled maintenance for Ajaxplorer (WebFTP) tonight @ 8pm PST (RESOLVED)”

All sites on our server lynx.dreamhost.com were hacked with Base64 code injections…have spent the last 3 hours trying to figure out what’s going on and fix…is this related to the post above? Help please anyone!

ps – still waiting to hear back from DH hack support…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘cayenne’ restoring from backups to new hardware (RESOLVED)
12:58 pm

Shared web server ‘cayenne’ restoring from backups to new hardware (RESOLVED)

Posted (February 21st, 2012 at 12:58 pm PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘cayenne’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘cayenne’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Feb 23rd, 2:05am PST: The restore on this server has completed and all sites are now loading from the new hardware! If you notice missing or outdated data, or experience any other issues, please be sure to contact support. – Chalise

Update Feb 22nd, 8:50pm PST: This server’s restore is continuing without any issues and services should all be online at this time. If you notice any issues with your hosting on this server, please contact support for assistance! -Dan J

Update Feb 22nd, 3:50pm PST: The restore process continues to execute without issue. If you notice any issues with your hosting on this server, please contact support for assistance! -Brian S

Update Feb 22nd, 4:25am PST: The restore process continues to execute without issue. If you notice any issues with your hosting on this server, please contact support for assistance! -JJ G

Update Feb 21st, 11:47pm PST: This server’s restore is continuing without any issues and services should all be online at this time. If you notice any issues with your hosting on this server, please contact support for assistance! -Patrick

Update Feb 21st, 7:33pm PST: All Apache servers are fully functional so your sites should be accessible as well. The restore is continuing smoothly and is currently at 21%. We’ll make sure to keep this post updated with its progress. Your patience is greatly appreciated. If you notice any issues with your sites, please contact support and we’ll be happy to help! -Seohee

Update Feb 21st, 4:36pm PST: The new hardware has been placed online. We’ve confirmed that all apache servers are fully functional. Therefor, your sites should be accessible as well. The restore has begun, and we’ll make sure to keep this post updated with its progress. Sorry for any inconvenience this has caused. Your patience is greatly appreciated. If you notice any issues with your sites, please contact support and we’ll be happy to help! -Brian S

Update Feb 21st, 3:16pm PST: The filesystem check is taking longer than expected to complete. Our admins have decided that the best course of action would be to move the server to new hardware. A NFS mount to our data backup machine will be created. This will allow your sites, services, etc to function as the filesystem check continues to run on the old machine.

We will update this post once the move has been completed. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for updates. -Brian S

Update Feb 21st, 2:10pm PST: The file system check is running without any issues. Once it has completed, your sites should come back up without any issues. In the event that we need to do a full failover we will update this post. Please continue to check back here for updates as the file system check continues. – Oscar

This entry was posted on Tuesday, February 21st, 2012 at 12:58 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

16 Responses to “Shared web server ‘cayenne’ restoring from backups to new hardware (RESOLVED)”

Having issues with reykjavik. Anything on that? Sites not loading, then working fine minutes later.

@Dave: This post is only referring to the server “cayenne”. “reykjavik” is fine though. Contact support if you are having an issue with your site and we can take a look.

Okay, thanks. I have noticed intermittent issues lately on all sites (multiple clients having same issues). I will submit a ticket. I was just curious if others were having issues on reykjavik.

So all my sites are on cayenne… [grumble]

I’m not usually one to post here, but this is more of a brain dump than a prompt for support. I don’t expect a reply, so I didn’t create a ticket.

It seems like in the last 6 months or so between slow sites, messed up databases, and downed servers Dreamhosts’ 99.9% uptime guarantee isn’t true anymore. Accordingly it seems like some pro-rating / customer satisfaction should happen.

So far I’ve been ok checking this page and waiting for your more than capable techs to fix the problem. But, I bring up your recent history and the subject of pro-rating cayenne users bills today out of building frustration with Dreamhosts’ customer service. Again, your techs are always on the ball. My frustration is the number of times I’ve had to contact them of late.

Regarding outages I realize that there are notifications that can be set, and an RSS feed on this page, but it seems to me that an entire server cluster going down warrants at least an email to notify users like myself that all my sites are offline. A simple machine generated email might have saved me the embarrassment of telling a room full of people what I can do with WordPress, and what Dreamhost can do for hosting, domains, and email, only to bring up site not found errors. I’m fond of the referral system you have in place and use it with as my of my clients as I can. Today however, this down time has made me, and you, look foolish. It cost us both business. It may yet cost you my business.

I don’t want this to be another ranting post from some kid on the internet, I’m actually concerned. So, what’s the deal guys? Did you get a new admin, new data center, different tubes — what’s going on at Dreamhost HQ that leaves a longtime, and fairly loyal, customer such as myself evaluating other hosts? I feel like there is a narrative behind all this that, if told, could ease customers such as myself. And to that point— you might ask yourselves what you can do, proactively, to smooth things like this out for your clients?

Frustrated,
~ Aaron

Any news on ps45477? I have been down for over an hour, chat support is down, I got an email from the VPS Stability Robot that said I needed to increase my resources, but I don’t think that is what is causing the problem anymore.

ps70863 down again.

Please note that this post is regarding the shared web server ‘cayenne’ and no VPS are affected. If you are having issues with an unrelated server, please do contact support and we’d be more than happy to assist you.

Dreamhost, why do you cap your 100% uptime guarantee at 10% of the next bill?

Since your guarantee is 1 hour of downtime = 1 free day, that means you only compensate 3 hours of downtime a month (assuming you pay monthly). Considering how often I see posts about servers being down for entire workdays, it makes the guarantee seem pretty halfhearted.

So my machine is not listed as having a problem. I contacted support an hour ago and still awaiting contact. This is happening WAY TOO often. Getting ready to leave for BlueHost. Unacceptable!

@Yolanda: I’m showing one of our support techs already responded to your support case and your site is up. If you need anything more feel free to reply to that email and we’ll help you out.

alouettes is down now

Warmachine also down right now

Is the servers down?

can anyone in dh check athens server? it has been quite slow since yesterday. thanks

@Michael – It looks like alouettes was restarted shortly after your comment was posted and has been running smoothly since.

@alfred – warmachine itself doesn’t seem to be having issues, but I do see there was an issue with your Apache service and your ticket was replied to.

@Laurie – No, cayenne is up and running and hasn’t had any further downtime.

@William – athens looks to be in great shape, but go ahead and submit a ticket in your panel under Support / Contact Support if you’d like us to look into any slowness you’re experiencing.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware ( RESOLVED )
3:52 am

Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware ( RESOLVED )

Posted (February 18th, 2012 at 3:52 am PST) by patrickm

Due to issues with the RAID controller on the web VPS host server ‘homie-vserver207′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver207′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:56pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Update Feb 22nd, 2:10am PST: This restore is progressing well and all VPS guests are reporting online at this time. Please feel free to contact support with any issues or questions you may have! -Patrick

Update Feb 21st, 8:48pm PST: All VPS guests are running and the server itself remains stable as the restore continues. We’ve got about a dozen VPSes left to restore. If you notice any issues with your VPS, please contact support and we’ll be happy to help! We’ll continue to post updates as the restore process completes. – Seohee

Update Feb 21st, 1:40pm PST: All VPS guests are running and the server itself remains stable as the restore continues. We are more than half way done with this restore. If you notice any issues with your VPS, please contact support and we’ll be happy to help! We’ll continue to post updates as the restore process completes. – Oscar

Update Feb 21st, 5:45am PST: All VPS guests are running and the server itself remains stable as the restore continues. If you notice any issues with your VPS, please contact support and we’ll be happy to help! We’ll continue to post updates as the restore process completes. – Chalise

Update Feb 20th, 7:51pm PST: All VPS guests are running and the server itself is stable. If you do notice any issues with your VPS, please be sure to contact support. We’ll continue to post updates as the restore process completes. – Dan J

Update Feb 20th, 5:10am PST: All VPS guests are up and running and the restore is continuing to run smoothly. About 70% of the VPS guests on this host machine have been restored, and we will continue to post updates as the restore process continues. – Chalise

Update Feb 19th, 6:50pm PST:  The restore is continuing, with all VPS guest machines up and running. The restoration process is going smoothly, data is currently being restored, and over a dozen VPS restores have completed. More updates to come as the restore progresses. – Seohee

Update Feb 19th, 7:00am PST:  The restore is continuing, with all VPS guest machines up and running. The restoration process has started, data is currently being restored, and over a dozen VPS restores have completed. More updates to come as the restore progresses. – Justin

Update Feb 18th, 11:15pm PST: All VPS guest machines are remain up and running. The restoration process has been started, data is currently being restored. More updates to come as the restore progresses. – JJ G

Update Feb 18th, 12:25pm PST:  All VPS guest machines are up and running on the new server using NFS mounted backups with the exception of one guest. Our admin team is working on fixing that VPS guest. Once all VPS guest machines are mounted, data will start to  be restored to each guest server one by one. More updates to come as the restore progresses. Thanks once again for your patience! – Oscar

Update Feb 18th, 5:17am PST: The new hardware is up and the VPS guests are being brought online at this time. Once all guests are live, we will begin the restore! Please check back here for updates! -Patrick

Update Feb 18th, 3:50am PST: We are beginning the process of swapping the hardware for this server, so services are offline at this time. We will update this post when the process is underway and VPS guests are back online. Thank you for your patience! -Patrick

This entry was posted on Saturday, February 18th, 2012 at 3:52 am and is filed under Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

38 Responses to “Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware ( RESOLVED )”

Affected VPS guests:
ps40175 ps40362 ps40490 ps40532 ps40897 ps41006 ps41069 ps41334 ps41347 ps41918 ps41973 ps42045 ps42529 ps43523 ps43809 ps43958 ps43984 ps46028 ps46305 ps46740 ps46921 ps47255 ps47273 ps47299 ps48931 ps48938 ps49258 ps50421 ps50454 ps50537 ps54055 ps54060 ps55435 ps55509 ps55787 ps56771 ps57473 ps59140 ps59395 ps59719 ps64523 ps65471 ps65487 ps65495 ps70863 ps71118 ps72475 ps72568 ps73323 ps73325 ps73634 ps74225 ps74707 ps74792 ps76359 ps83442 ps85258

Our site has been down since yesterday again, this is unreal. I have submitted three tickets and no response from you. I am at wits end on this because we are loosing a ton of money and business. I expect compensation and some sort of explanation immediately and no “month” of free hosting is going to be enough. This has happened multiple times and I have a ton of other websites hosting with other companies and not once have I had issues like this. Especially with NO notification! Please respond to me ASAP about this issue with both of our sites. I have bosses yelling at me about this!!

Well, I have been very sympathetic and supportive over the last 90 days as Dreamhost suffered through some obviously tough times. Issues happen, that is understandable.

Here is my question/issue, however… Are you (Dreamhost) not aware of the administrative user for the pool of VPSs that you are working on? Is it not a reasonable tactic to message that user that their VPS is in this pool, ergo their services are down? When I log into the Dreamhost panel, I see no notification. When I check my email, no notification. When I am puzzled about my front end services being unavailable, and visit manage resources, I see no flag next to that VPS.

Problems happen, big whoop. How you deal with them differentiates providers. Maybe I am missing something, but flagging my account so at login i see a notice, emailing me, etc does not seem all that challenging. For me to have to kill an hour scratching my head, then going to status.dreamhost.com, then having to crossreference my VPS name to confirm it is in this pool is just kind of silly, don’t you think?

What a joke. We’re down and not even the list of affected servers above. Last time this happened we were down for more than 48 hours straight. I’ll be finding another provider this afternoon.

My web site has been experiencing problems for several weeks now. The first time they discovered a fault with the server software not closing connections and therefore people had to wait 2-3 minutes for simple pages to download.

The problem has recurred three times since. The Support people have generally been helpful.

However, today things went wrong yet again at a crucial time. I have been trying to get the support people to reply to my support requests for five hours – without any response or even acknowledgement.

I appreciate that Dreamhost offers a good deal financially – but that’s where the good deal ends. The quality of the service is downright poor and now the support has become non-existent.

The worst thing for any customer to be faced with is unreliability – and Dreamhost seems to be guilty of this on all aspects of their service – machine and human.

This is just unbelievable. Over 24 hour failure, twice in less than a month. While my websites are running on the backup version, they do not reflect the latest changes and I cannot upload anything. Chat support is dead. I need another solution asap. Anyone?

This is what makes me smile:-

“Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”

It seems there is no proper way to contact them as they don’t reply to tickets. lol

F*ck you Dreamhost and your f*cking BS service.

So I guess I am the lucky ‘guest,’ because my VPS is still a dead stick… 14 hours and counting.

All site down, server ps64842

Day five of mostly dead site and no FTP access for me. I’ve loved DH for years, but it’s really time to move on now :(

This is crazy!! I’ve had to sit and explain to our readers what’s been going on and you’ve affected my business greatly this month. Nowhere near where we should be thanks to the downtime. Free domains just isn’t enough. The downtime has cost me a lot more.

@Marie: Sorry that the various issues this year have affected you. All services should be up at this time, if you are still having issues, please submit a support ticket or provide a domain that isn’t working for me to find your account and investigate.

@Vincent: Your VPS is being restored at this time but is on homie-vserver197, a different status post. I tested and was able to load some randomly picked sites. If you see any in particular that are not working, please submit a support ticket with the details.

@Michael Seguin: I’ll take a look at your VPS now to see if anything can be done to get it on local disk faster.

@Logan: Tickets should be replied to within the usual 24 hours maximum. During a move to new hardware like this though, unfortunately some wait time will be necessary to get services working again.

@Fred: We have access to the old hardware and can restore data if needed. Please submit a ticket and we can get any missing data restored.

If I missed anyones issue that hasn’t been addressed, I apologize, please report here or via a ticket and I would be happy to assist where I can.

VPS guest: ps42045

I’ve submitted a ticket a few hours ago with no response from the support team. Can someone please get back to me?

I have a vps elesewhere for a long time and never a single issue. Yet with DH my vps, shared accounts are constantly stuffed around. This really is poor service. And yet they are happy to keep billing our cc’s

what happened to livechat? are those people all working on the issue/s

ps70863

Looks like I will only have to fulfill three web orders over this weekend. Slowest weekend in three years of doing business on the web.

Still down.

Site in question: justmymusic.com — and yes, I do have a web ticket open. Guest ps70863

$128.00 bucks to Google for no actual visits to the site since it is down. Ouch!

@jasmel: Sorry for the issues. The unique ip was not properly setup on the VPS, for that site to work. It is working now after configuring and restarting the VPS and apache services. We can’t monitor every individual site for proper function unfortunately, and the apache was up and running so it wasn’t reporting to in our monitoring that your VPS was having issues. I am answering your ticket now that was submitted early this morning.

@jimbo: You didn’t provide any details such as your VPS number or a domain to look up your account. The only open issue currently as for VPS restores, there are no known shared server issues at this time. If you wish to provide details in a support ticket we would be happy to look at your situation and move you to other servers, etc if it would improve your experience. As for livechat, we hope to offer it again in the future, but currently cannot provide it with the increased support volume. We are working on fixing that situation.

@Bob: The VPS has around a 2 day uptime, although I was seeing a session error at the top of that page. I got it fixed up, it should be loading without that error now.

My site is also down. Completely white screen. It is not listed in the affected servers above and I have yet to hear back from support. I don’t know whether to run a restore or not. I have loads of patience from DH but there comes a breaking point…A little more feedback would be nice.

I did notice recently an odd uptick in my memory usage the last few days, including a distinct, short-term spike on the 18th requiring a reboot. I understand these things happen, and am providing this information to assist you. My machines are ps76062 and ps41073.

live chat dead, ticket support not responding, cannot afford it anymore. Need to switch to a better service with 24/7 live phone/chat support.

@Fred: Please provide a VPS server name (psNUM) or a domain and I can look up your issue right now. I do not see any tickets open by anyone with the first name of “Fred” currently.

@Steven Savage: Neither of those VPS are on a vps host machine that is restoring from backups currently. Spikes in traffic and/or memory can come and go. Have you analyzed your traffic to see if more search engines hit you that day or other kind of traffic?

@Paul: You are not affected by any issues on this status blog. The joomla install you are running is outdated, please consider upgrading.

@Justin – I’ve considered upgrading but it is not an option. It was working perfectly fine previously and not has a blank front page now. I can get into the backend OK and I can ping the website.

When I go directly via IP address I get the following error message:
Site Temporarily Unavailable

We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.

error id: “bad_httpd_conf”

This is something that I have had previously and was resolved fairly quickly via support. I hope that will happen again, because as I said, upgrading is not an option.

Thanks

@Paul: Upgrading is extremely important, if you do not, you are putting your site at risk of being compromised, which may entirely be the cause of the issues you are seeing. Running a 2 year old joomla install is highly discouraged. It is also normal and expected to see a bad_httpd_conf on shared hosting when viewing via ip, we do not allow viewing of a site via ip alone except if a unique ip is used.

@Justin I can’t even run a backup. I have tried twice now and nothing. I had an email from support to say my site has been attacked and that it was being referred to the security department but nothing since. None of my file appear to have been modified. The last time this happened I got a canned response about things to do. Not everyone who used DH are as technical as the people being the scenes. I have an important charity event advertised for registration. With the site being down no-one can register and we are running out of time. Right now I need the site up and running like it was yesterday. I am in the process of a new site but I am not ready to move it across yet.

There was a time when a support ticket would be dealt with in a speedy manner. The customer service experience has been lacking lately and not only myself but others are suffering.

@Paul: I don’t recall seeing any open tickets after I wrote to you, if you wish a security scan please open another ticket and ask for a security scan if you wish. The first thing our abuse team will say after scanning for a compromised site is to upgrade ALL software, so if you are unwilling to do so, I am not sure what more we can do for you. The software is free, open source, and there are known vulnerabilities in the version you are running that is (I believe) 2 years old!

@Justin – This is the mail i received:

I apologize for the inconvenience. I am going to forward your ticket to
our security team as it seems your site was attacked and the result is
this white page. I am sorry for the trouble, the security team will
follow up with you and be able to provide more information.

Thanks!
Matt S

There was no reason for me to open a new ticket. I am still waiting to hear from the security team although I suspect this will be tne normal canned response. I have also received an email saying the following:

Hello,

Unfortunately we were unable to process your domain restore for
jaylenschallenge.org as there were no available backups!

We are very sorry about this! Howevvvvver… we actually make no
guarantees about availability of backups, and highly recommend you
always keep your own copies of all important data. Please follow the
link below for instructions with this:

http://wiki.dreamhost.com/Backup_Website_Manually

Thanks,
The Unhappy Dreamhost Domain Restore Robot!

Do you really think it is the time and place to add a little sarcasm to your customers? I have always enjoyed the relaxed nature of dreamhost in the past but with my back up agaisnt the wall it doesn’t seem very funny anymore.

I would also like to state that the case isn’t about me being unwilling, but about me not being able to at this moment in time. You say you are not sure what else you can do…well how about fixing the backup system so that It doesn’t take several hours to tell me it failed.

@Justin – I have it working now. Thanks for all your help.

@ Justin. I have never been able to issue a ticket. When I go to the chat live, it says that chat is currently offline and I that I should submit a support ticket. Just underneath, there is a “submit” button. When you click it goes nowhere. If this button is not to submit a ticket, what is it for?

@Fred: In your panel, go to Support / Contact Support, here:
https://panel.dreamhost.com/index.cgi?tree=support.msg&

Scroll down to “Select the type of problem you’re having”

Select the issue, and click the blue “next” bottom to the bottom right. There will be a series of steps, including a box to type up your issue and submit the ticket. The panel will confirm when the ticket has been submitted.

My site is down, balamoosh.com on fomalhaut server :/

“Your website does not appear in the Apache configuration file.”
“Your home directory does not exist on your web server.”
“Your website took longer than 5 seconds to respond.”

When it says, “All VPS guests are running and the server itself is stable,” does that mean that all the sites are up? Or does a site not get back online until it is restored?

@Aaron: It means that the sites are up and running from backups mounted to the new server. When the restore is done, we will unmount the backups and then your sites will be running off the files on the new server.

ps70863 — down again. Ticket submitted but not sure what else to do but wait and pause my Google Ad Words yet again.

@Jasmel: Very sorry about that. I had an administrator look into the issue and he’s got your VPS back up and running. I’ll send you a response to your open ticket with the details.

Support ticket in. How hard is it when migrating to new hardware to leave the old site alone until thorough testing is done and then flip the switch? I get emails that it is working, but it is not. I think it has been down 3 days now…pretty unbelievable and yes I have several support tickets going. Don;t know if you all think it is working or not. PS74792

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘paramaribo’ offline for RAID card replacement (RESOLVED)
4:31 pm

Shared web server ‘paramaribo’ offline for RAID card replacement (RESOLVED)

Posted (February 17th, 2012 at 4:31 pm PST) by Dan J

We apologize for the inconvenience, but we had to take the shared web server ‘paramaribo’ offline briefly for a RAID card replacement.  We expect this maintenance to take approximately 20 minutes.  We will update this post when the issue has been resolved!

Please note, this is regarding the shared web server ‘paramaribo’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Feb 17th, 4:50pm PST: The RAID card has been replaced and this server is now back online. If you continue to have issues then please contact support. -Dan J

This entry was posted on Friday, February 17th, 2012 at 4:31 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver173 offline for file-system check (RESOLVED)
4:03 pm

Web VPS host homie-vserver173 offline for file-system check (RESOLVED)

Posted (February 17th, 2012 at 4:03 pm PST) by Dan J

The web VPS server homie-vserver173 is offline for an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We expect this process to take about two hours and we will provide frequent updates below as the check completes. We apologize for the inconvenience that this situation causes!

We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

A list of the affected VPS guests can be found in the first comment below this post.

Update Feb 17th, 4:50pm PST: The file-system check has completed and this server is now back online. All VPS on this machine should be online as well. If you continue to have issues then please contact support. -Dan J

This entry was posted on Friday, February 17th, 2012 at 4:03 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Web VPS host homie-vserver173 offline for file-system check (RESOLVED)”

Affected VPS guests:

ps37393 ps37419 ps37462 ps39142 ps39159 ps39162 ps39381 ps39388 ps39414 ps39418 ps39423 ps39561 ps39569 ps39584 ps39947 ps39975 ps40397 ps40685 ps40898 ps41068 ps41072 ps41422 ps41510 ps41623 ps41903 ps42001 ps42303 ps44457 ps44583 ps45065 ps45632 ps45690 ps46094 ps46106 ps46127 ps46554 ps46791 ps46956 ps47188 ps47357 ps47728 ps48057 ps48081 ps49327 ps50502 ps52416 ps52432 ps53147 ps53373 ps54114 ps55484 ps55485 ps56454 ps56458 ps57506 ps68922 ps71111

tgif! But website is still down.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver197 restoring from backups to new hardware ( RESOLVED )
3:09 am

Web VPS host homie-vserver197 restoring from backups to new hardware ( RESOLVED )

Posted (February 17th, 2012 at 3:09 am PST) by chalise

Due to issues with the RAID controller on the web VPS server ‘homie-vserver197′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS server ‘homie-vserver197′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

A list of the affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:58pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Update Feb 22nd, 2:01am PST: Just a few more VPS guests to restore and then this process will be complete! All the guests are online at this time, so please contact support with any issues or questions! -Patrick

Update Feb 21st, 8:06pm PST: We are still waiting for the  final few VPS guests to finish their restores. All the VPS guests on this host machine are running and the server itself is stable as well. If you notice any issues with your VPS, please contact support. More updates to come as the restore process completes. – Seohee

Update Feb 21st, 1:50pm PST: We are still waiting for the  final few VPS guests to finish their restores. All the VPS guests on this host machine are running and the server itself is stable as well. If you notice any issues with your VPS, please contact support. More updates to come as the restore process completes. – Oscar

Update Feb 21st, 5:55am PST: We are even closer to completion as the final few VPS guests finish their restores. All the VPS guests on this host machine are running and the server itself is stable as well. If you notice any issues with your VPS, please contact support. More updates to come as the restore process completes. – Chalise

Update Feb 20th, 7:49pm PST: Only a few VPS guests remain before this restore is complete! All VPS guests are running and the server itself is stable. If you do notice any issues with your VPS, please be sure to contact support. We’ll continue to post updates as the restore process completes. – Dan J

Update Feb 20th, 7:49pm PST: Only a few VPS guests remain before this restore is complete! All VPS guests are running and the server itself is stable. If you do notice any issues with your VPS, please be sure to contact support. We’ll continue to post updates as the restore process completes. – Dan J

Update Feb 20th, 5:20am PST: The restore process is continuing to run, with just a handful of VPS guests left to restore. As always, if you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! – Chalise

Update Feb 19th, 7:33pm PST: The restore process continues and there are just about a dozen guests left. If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! – Seohee

Update Feb 19th, 7:30am PST: The restore process continues and is at around the half way completed point. If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! -Justin

Update Feb 18th, 11:00pm PST: The restore process continues to execute and the server remains stable! If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! -JJ G

Update Feb 18th, 12:30pm PST: All issues with the VPS guest machines on the new server have been fixed. The restore process has continued and all VPS guests are online at this time! If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! -Oscar

Update Feb 18th, 6:41am PST: We are aware of the issues with some guests on this server and our team is working to resolve those problems right now. Otherwise, the restore is continuing. We will post again shortly with a resolution to the current downtime. As always, please Contact Support for more assistance! -Patrick

Update Feb 18th, 1:21am PST: The restore process is moving along well and all VPS guests are online at this time! If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! -Patrick

Update Feb 17th, 8:25pm PST: The restore process is running well. We will continue to update this post as the process moves along. Your sites should be accessible as the restore is happening. If you see problems with your service, please let us know via our Support System. Thanks, once again, for your patience! –Dan J

Update Feb 17th, 2:32pm PST: The restore process has begun. We will continue to update this post as the process moves along. Your sites should be accessible as the restore is happening. If you see problems with your service, please let us know via our Support System. Thanks, once again, for your patience! – Brian S

Update Feb 17th, 12:25pm PST: There is one VPS guest which is causing some issues on the new server. We cannot start the restore until this issue with that VPS is fixed. We will continue to update this post as soon as we have more information. Thanks once again for your patience! – Oscar

Update Feb 17th, 10:10am PST:  At this time 90% of VPS guest machines are up and running on the new server using NFS mounted backups. Once all VPS guest machines are mounted, data will start to  be restored to each guest server one by one. More updates to come as the restore progresses. Thanks once again for your patience! – Oscar

Update Feb 17th, 8:10am PST:  The replacement hardware has been setup by our admin team and we are starting to mount all VPS machines via NFS. Once that is done all VPS servers will be running off backups.  Around 20% of the VPS guests have been created and the rest are being created as fast they can. Once this is to 100%, all VPS should be hosting sites normally and the restore from backup can continue in the background. We will continue to update this post again as the restore progresses. Thanks for your patience. -Oscar

Update Feb 17th, 4:20am PST: The file system check is being monitored closely and is continuing to run; it has finished its first pass with a few cleanups and is restarting. We will update this post again shortly with more further information. Again, if you have any questions or concerns, please be sure to contact Support under Support > Contact Support, in your panel. – Chalise

This entry was posted on Friday, February 17th, 2012 at 3:09 am and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

42 Responses to “Web VPS host homie-vserver197 restoring from backups to new hardware ( RESOLVED )”

Affected VPS guests:

ps11126 ps11137 ps11156 ps11277 ps11323 ps11594 ps12547 ps14026 ps14044 ps14047 ps14049 ps14050 ps14055 ps14107 ps14117 ps14236 ps14471 ps14483 ps15218 ps16414 ps16995 ps18375 ps18445 ps18447 ps19297 ps22433 ps22758 ps24132 ps27946 ps36926 ps36957 ps36960 ps36972 ps36986 ps36990 ps38891 ps58381 ps59864 ps61109 ps62047 ps62592 ps64822 ps64842 ps65022 ps65038 ps65079 ps65626 ps6821 ps6822 ps6825 ps6827 ps7006 ps70482 ps70501 ps7176 ps7377 ps7421 ps74229 ps7615 ps7864 ps8305 ps8710 ps8828 ps9038 ps9501 ps9544

Any updates on this?

Anything?

Yes, once again my VPS site is down for hours. Thanks for “frequent updates” too

nothing but crickets. Unbelievable!

*chiiiirrrp chiirrrrp*

How’s the restore progressing?
How about posting a list of VPS servers as they come back online.

how about having a way to contact you instead of this stupid blog.
we area also down for hours, can not get through the support channel, of course you will not publish a phone number
this company is a complete Joke

So much for the 2 hour estimate at 3am

What is the chat offline?

Thanks for getting us off the black lists, but now our website is down. HELP.

Another entire day down. Come on guys!

UGH! UGH! UGH! UGH!

ERROR ERROR ERROR

What is going on?

Anybody?

OK, what’s going on?

We are dead in the water.

MAJOR FRUSTRATION

Imagine that yet another issue with dreamhost. Seems it’s one thing after another. Can you not have an issue for one month? Getting sick of the poor service..

Problem is NOT resolved. Our site came up for 20 mins and went back down. We r down presently.

So our sites should display?

Update Feb 17th, 2:32pm PST: The restore process has begun. We will continue to update this post as the process moves along. Your sites should be accessible as the restore is happening. If you see problems with your service, please let us know via our Support System. Thanks, once again, for your patience! – Brian S

so much for this…

Enterprise-Level Reliability

DreamHost manages its own network, so that your site never skips a beat. DreamHost’s network is backed by a 100% uptime guarantee policy and you’ll have around-the-clock (24×7) access to our expert technical support representatives via email, and live chat.

why is our site still down?

There has been no live chat in at least the last 5 weeks. Maybe my timing is off.

“Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver206′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. ”

Why the heck couldn’t this be done with THIS server before pulling the plug?

My clients are so outta here. Done.

Status?

Hello?
Anyone?

Please clarify latest post. Should all the vps servers affected be able to see their sites right now? I can’t.

My site is still down, when are things coming down to normal? Do we get a refund?

NOPE, I’m still down and did contact support

Update Feb 18th, 1:21am PST: The restore process is moving along well and all VPS guests are online at this time! If you notice any issues with your hosting on one of the affected servers, please Contact Support and we will be happy to help! -Patrick

Still down!

over 24 hours and STILL DOWN. this is unacceptable.

Still down. No recent updates

Last update is very generic.
I am still down and assume I’m part of the “some guests”.
I contacted support and opened up another ticket.

Update Feb 18th, 6:41am PST: We are aware of the issues with some guests on this server and our team is working to resolve those problems right now. Otherwise, the restore is continuing. We will post again shortly with a resolution to the current downtime. As always, please Contact Support for more assistance! -Patrick

The first post lists affected VPS guests.
Maybe another list of who is up and who is still being worked on.

“…all VPS guests are online at this time.”

Negative, Comrade.

@morgan: We do not recommend contacting us via the blog, but instead via a ticket. A ticket can be submitted via the link in the status post (preferred), or via the contact us link on the dreamhost.com page. The website you linked in your comment is loading for me at this time.

@Dana Bauguess: We are also glad the blacklist issue is looking better. The site you linked in your comment is loading at this time, if you still have issues, please let us know via a support ticket.

@Madge: The site you linked is not hosted on a VPS and therefore not related to this issue. I am sending you an email with some tickets to reduce usage so your sites will have less issues.

@WTF: It couldn’t have been done before the server failed, because it hadn’t failed at that time. We are working on improving raid reliability and monitoring, for future improved stability.

@Jill: Working on your VPS right now.

@Vinny: I can look for your tickets if you still are having issues, if you can provide the name of the VPS or a domain.

@Michael C: Looking at your domain, which is currently showing a bad_httpd_conf error.

@Justin: Thank you.

FYI, neither of the 2 support tickets I’ve submitted over the past day have been acknowledged in any way other than being logged in the panel. I have received neither human nor any automated email responses to them. Any clue what _that’s_ about? If nothing else, I’d at least have expected to receive the automated “we got your support ticket” email, but nada.

@Michael C: I fixed up your VPS, it is on local disk and no further changes will be needed during this restore process.

@Jill: Your VPS should also be working normally now.

@Justin: Got your mail stating that you got my sites back up, and indeed, they are. I wanted to thank you publicly, so… thank you!

@Michael C: I found the two tickets and responded. For normal support emails not to a specific person (if they provide you such an address, I think I provided you mine for further issues) it generally doesn’t send a confirmation email.

@Michael C: You are very welcome. There still can be a bit of slowness now and then until the restore completes, but it should be greatly improved.

add ps70863 to that list.. site is down.

@jasmel: Sorry for the issues. The unique ip was not properly setup on the VPS, for that site to work. It is working now after configuring and restarting the VPS and apache services.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mailservers on Trend Micro RBL, working on removal from list (RESOLVED)
12:54 pm

Mailservers on Trend Micro RBL, working on removal from list (RESOLVED)

Posted (February 14th, 2012 at 12:54 pm PST) by brians

We’re well aware of the email issues that you may be experiencing. Unfortunately Trend Micro, maintainer of an RBL (real-time blacklist,) has blacklisted several of our mail servers. Customers of service providers who subscribe to this RBL will not be able to receive email from DreamHost customers until this is resolved.

After several days (nearly a week) of relentlessly trying to get a response from Trend Micro we have finally heard back from them and are processing through a list of their concerns as quickly as possible.

We are hopeful that when we resolve these outstanding issues Trend Micro will take action in a timely manner – and mail service to the affected service providers should then resume.

We apologize for the inconvenience this has caused and appreciate your continued patience. -Brian S

Update March 1st, 5:33pm PST: The mailman server has been unblocked as well, so we’re resolving this case. If you run into any mail blocks, please submit a ticket through the DreamHost panel including the full bounce message with the date and time, so we can address it. – Andrea

Update Feb 28th, 5:33pm PST: We are continuing to get this case fully resolved. We are at the brink of getting that one discussion list unblocked. We appreciate your continued patience. – Andrea

Update Feb 27th, 12:42pm PST: Unfortunately, there is no change in the status below, just letting you know we’re still working on it. We appreciate your continued patience. – Andrea

Update Feb 26th, 2:15pm PST: One of our discussion list servers and our autoresponder server is still blocked. We’re continuing to cooperate with Trend Micro to get this resolved, and working on further reviewing our outgoing mail system to avoid potential issues in the future. We appreciate your patience. If you’re continuing to see issues unrelated to a discussion list or an autoresponder, please submit a ticket through the DreamHost panel including the full bounce message with the date and time. Your case may be unrelated (other block providers), but we would like to take a closer look. – Andrea

 

Update Feb 23rd, 11:24am PST: The block has been removed from the announcement list server last night, so those messages should be going through at this point or shortly. Unfortunately, some of the discussion list servers continue to be affected, and so are the autoresponder messages.
To repeat: no email sent from regular email clients and web server (directly or as part of a web application) should be affected at this point. If you get a bounce message after sending through any of the latter ones, please submit a ticket through the DreamHost panel including the full bounce message with the date and time.

We apologize for any inconvenience this is still causing – Andrea

Update Feb 22nd, 11:58am PST: All e-mail servers have been removed from the RBL, meaning that e-mail client sending functionality should still be working without issue.

Our relay machines are also no longer on the RBL, so web based e-mail ( ex. php mail(), sendmail, etc ) should be successfully delivered. Please note that there may still be some delay in removing the block from your specific ISP.

The machines which handle discussion lists, announcement lists, and auto-responders are still currently listed on the RBL. We’re continuing to work with Trend Micro to get those removed, and hope to have them de-listed soon. Thank you for your patience in this matter. -Brandon

Update Feb 20th, 1:09pm PST: We’re happy to report that the block has been lifted from all email servers. This means, that if you’re sending mail through an email client, it should be successfully delivered.

Our relay machine has been unblocked as well. Therefor, mail originating from our webservers ( ex. php mail(), sendmail, etc ) should be successfully delivered. Please note, even though the block is gone, it takes some time to propagate through all providers. This can take up to 24hrs but likely less.

There is still a block on the machines that handle our discussion list, announcement list, and auto-responders. We’re continuing to work with Trend Micro to get those removed as well. We appreciate your continued patience. – Brian S

Update Feb 16th, 3:30pm PST: We’re happy to report that the block has been lifted from all email servers. This means, that if you’re sending mail through an email client, it should be going through soon. Please note, even though the block is gone, it takes some time to propagate through all providers. This can take up to 24hrs but likely less.
Those of you sending mail through web servers may still experience the block. We’re continuing to work with Trend Micro to get those removed as well. We appreciate your continued patience. – Andrea

Update Feb 16th, 10:56am PST: Some of you may be receiving messages from Trend Micro when contacting them directly about these blocks stating that we didn’t take action after being made aware of the spamming from our server. We would like to assure you that we’re continuously working on processing all the claims they are sending our way, so you may be receiving canned messages that don’t apply to this case. Very sorry about the delay in getting this resolved. -Andrea

Update Feb 15th, 10:11am PST: We’re continuing to work directly with their higher level of support to get this issue resolved through cooperation. We’re also working out the details for working together in the future to avoid this situation from happening again. For those of you asking to be moved to a different mail server, unfortunately that wouldn’t make a difference. They are blocking the majority of our load balanced relay servers. We appreciate your continued patience. – Andrea

Update Feb 14th, 4:55pm PST: We have finished processing all of their concerns and communicated that to them in a timely manner. We are continuing to work aggressively on getting the Trend Micro block removed but haven’t received an ‘all clear’ yet. Very sorry about the inconvenience this has been causing. Please check back for updates. – Andrea

This entry was posted on Tuesday, February 14th, 2012 at 12:54 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

100 Responses to “Mailservers on Trend Micro RBL, working on removal from list (RESOLVED)”

Please list the major service providers that subscribe to Trend Micro. This will let us know who is not getting our outgoing mail.

I have been waiting for some replies from clients – now I think I know why I have not received them.

February 14th, 2012 at 1:25 pmopensourceeditor Says:

Why, WHY for christs sake is ist so hard for dreamhost to inform their customers via mail?
I have to read about every this and then scenario after i’m logging into my c-panel.
I WANT TO HAVE SUCH INFORMATION IN MY MAILBOX DAMMIT!

@Alan Stout –

How would DH know who subscribes to Trend Micro’s services? DH hasn’t blacklisted the servers, and I doubt that Trend Micro is willing to give up a list of their major subscribers.

@opensourceeditor –

First, DH doesn’t have ‘cpanel’. The DH panel is a custom application. Second, you can easily subscribe to the RSS or Twitter feeds of this page, which give you instant notification when something is posted here. Email is impractical for many reasons, not the least of which is the sheer number of customers that DH has (do you know how long it takes to queue and send > 800k emails? I’d much rather have them post here once than clog up their uplinks sending emails to customers). And doesn’t attempting to inform people about an email issue via email seem a little stupid, if you think about it? The whole reason why they have dreamhoststatus.com is so that you DON’T have to log in to anything to see what is happening.

February 14th, 2012 at 1:37 pmopensourceeditor Says:

and you are lying
“After several days (nearly a week) of relentlessly trying to get a response from Trend Micro we have finally heard back from them and are processing through a list of their concerns as quickly as possible.”

Here’s what the RBL-Team wrote back (at 5PM MEZ) 20 Minutes after i have issued a blocked IP warning to them via their Contact Form, their reply says among othe things: “no action was taken when we reported this to your ISP”

WTF!?

@opensourceeditor: It can take several days to send out an email to every one of our customers. That is true with our newsletter as well, and to email all of our customers about an email block issue, would likely be counter productive to getting unblocked. We use this status blog for more immediate updates for this reason. Those customers who have written in about this issue, are being updated via messages in our support system (emails).

@Alan: It is not really easy to give a list, as anyone is allowed to use the RBL.

I agree with opensourceeditor that you really should email us about these kinds of things. I had 2 clients of mine comment on this to me this morning, and I just shrugged it off as bad email addresses.

Only when I have logged in, I get to see it’s the case.

In future, please let us know what is going on via email as well. We seem to get 99% of the items via email. I would never have thought to login to determine if emails are being blocked.

Thank you

@opensourceeditor: We have heard that same claim from support from the RBL list, but it is incorrect.

This is the response I received within minutes of my doing an IP lookup and request to be unblocked and replying to the first canned reply email with a request for a real answer.

You are the only ones that can fix this!

We need it fixed…………pleeeeeeze !

Hello,

Thank you for contacting us about this IP address on RBL. This IP is listed because we received spam from this address and because no action was taken, when we reported this to your ISP.

We need to hear from your ISP prior to removing any part of this listing. Please secure your network and contact your ISP, and have your ISP contact us for removal. Any removal request of this IP must come from them or any authoritative administrator of this IP, so you may just invite your ISP to contact us at rbl@mail-abuse.com to send their removal request.

Kind regards,

Spam Investigator Engineer
Trend Micro Inc.

After several days of not reacting to support tickets on this issue finally an announcement.

TrendMicro might be slow in their response, but that’s not an excuse to keep quite for nearly a week.
In the meantime you could provide some relay services to allow your customers to send mail.

The TrendMicro thing is just one RBL we are listed on. We are listed on 6 other RBLs.

Backscatter.org for example:

“Please note that this listing does NOT mean you are a spammer, it means your mailsystem is either poorly configured or it is using abusive techniques”

Need some relief, please.

February 14th, 2012 at 1:44 pmopensourceeditor Says:

“First, DH doesn’t have ‘cpanel’”

shortcut for “Control Panel

“And doesn’t attempting to inform people about an email issue via email seem a little stupid, if you think about it?”

gmail works, i’ve tried it out. that’s wha i have a separate e-mail.

I do not want to set up a fight here. Server Crashes, Downtime, even if it keeps me from working, there are some things the can be forgiven but I’m mostly pissed about what the RBL-Team wrote me back today.

@opensourceeditor: That reply that Trend Micro sent you is misleading. It may be a generic reply that they use in situations like this, but we have been in contact with them since we first noticed the block last Thursday.

February 14th, 2012 at 2:00 pmBenjamin Kraus Says:

Two questions:
(1) Which of DreamHost’s IPs are being blocked? In other words, short of waiting for someone to both notice they didn’t receive an email from me and then tell me, how do I know whether my service is being affected? Is it as simple as doing a DNS lookup on mail.mydomain.com, and then checking the list for that IP address?

(2) If you have been “relentlessly trying” for “several days (nearly a week)”, why is this the first time it has been posted to DreamHostStatus? I understand from your comments above why you don’t notify everyone of this issue via email, but that doesn’t explain why it wasn’t posted to this blog “several days” ago.

– Ben

February 14th, 2012 at 2:11 pmopensourceeditor Says:

“but we have been in contact with them since we first noticed the block l a s t T h u r s d a y.”

:/

I cannot believe what you’re writing here

does this mean if we move and change dns that those emails are blacklisted or something like it?

Hey Peeps, I have been in contact with Dreamhost staff since last Friday on this very issue. They have been very responsive and I could tell even Monday the frustration they were having then with no responses from Trend Micro.

Thanks DH guys, for those of us who manage our clients and use the ticket system, still remain well informed, and we’ve always got a response within the first 24 hours (as typical with our 6 years at DH).

If it were in their control, I am sure this issue would have resolved last week. From what I can tell it only affects Optus customers in Australia for our clients. Changing their SMTP server to their Local server, resolves this for us :)

Good work DH, how slack that its taken so long for you to get a response from Trend Micro.

I was just moving away from my gmail account due to google t & s I disagree with, and now it seems I better keep it, as my domain mail won’t arrive.

This whole problem takes a bit too long. My mails did not arrive and I got into trouble because of that.

I know DH TEeam works on it day & nite, but this really hurts.

anybody an idea how to check what your email server IP is and how to check for black listing?

This, folks, is an unfortunate consequence of having hundreds of thousands of customers using an emial system. DH isn’t the only host with this occasional problem. Some time back, I remember when 1and1 had to cope with blocking of some of their IP numbers, not to mention HostGator and other hosts. It’s something they just have to work through, but remember that they didn’t create the problem. It was caused by customers who abused the system and sent spam. When that happens, the majority of people who follow the rules have to suffer, unfortunately.

Peace,
Gene

Like Erwin, I’d like to check to see if the IP assigned to our mail server is on the blacklist.

However, when I try to use a whois service to check mail.whatever.com I get an error. I guess they don’t like the subdomain. So is there a way to discover the IP to check against the blacklist?

-Bill

Never mind. It just occurred to me that I could ping my mail server and the address it is using is displayed.

Message I received TODAY from rbl@mail-abuse.org
————————————————

juan bartolome via RT rbl@mail-abuse.org

4:39 PM (3 hours ago)

to me
Hello,

Thank you for contacting us about this IP address on RBL. This IP is listed because we received spam
from this address and because no action was taken, when we reported this to your ISP.

We need to hear from your ISP prior to removing any part of this listing. Please secure your network
and contact your ISP, and have your ISP contact us for removal. Any removal request of this IP must
come from them or any authoritative administrator of this IP, so you may just invite your ISP to
contact us at rbl@mail-abuse.com to send their removal request.

Kind regards,

Juan Miguel N. Bartolome
Spam Investigator Engineer
Trend Micro Inc.

——————————————– Managed by Request Tracker

Dreamhost’s mail servers are all over the blacklists. Do this: Check any of the dreamhost IP addresses on this great “overview” blacklist site:

http://whatismyipaddress.com/blacklist-check

for example: 208.113.200.5
or: 208.113.207.5

They come up with MULTIPLE blacklists. We have CONTINUAL problems sending mail into Europe and to various major US manufacturers because Dreamhost can’t keep their AUTHENTICATED e-mail servers clean.

@Gene

“This, folks, is an unfortunate consequence of having hundreds of thousands of customers using an emial system. DH isn’t the only host with this occasional problem.”

I agree except it isn’t an “occasional” problem. This has been happening for over a year. I’ve been with DH for 10 (!) years and this has been growing worse and worse.

Is anyone else having trouble with Comcast blocking? We’re not getting the mail bounced, it just isn’t being received. It’s like it goes into a big black hole and doesn’t come out. Is it part of this issue or another block that will need to be dealt with separately?

Some of the blacklists don’t bounce – they just suck it up. But check spam filters too.

This is a prime example of why RBL are the absolute worst method of spam blocking. By blocking Dreamhosts mail server, Trend Micro has turned ever legitimate email Dreamhost customers send into a false positive blocked spam result.

The best part is that once Dreamhost was blocked, the spammer moves on to a new provider and goes on spamming immediately with minimal interruption while Trend Micro sits on their hands allowing all the legitimate email to be blocked for weeks at a time. And on top of that, every rejected email generates a response email which probably causes more wasted mail load than the spam it saved.

Then there’s the wasted man hours dealing with this crap, the lost income from lost customers and lost sales, and the 800k emails Dreamhost had to send to their customers to let them know what’s up.

In short, if you run RBL then I hate you immensely.

I’ve made this suggestion to DH before:

How about maintaining more than one “level” of mail server within DH. The top level could perhaps be reserved for long-time trusted DH customers and have serious restrictions (limited email volume, no mail lists at all, no PHP script emails, etc.) “Expert/Trusted” users.

Lower levels could be assigned to more risky customers: new customers w/o a long history with DH, or customers that want to run large mailing lists.

Lowest levels would be for brand new customers, PHP email scripts, etc. : stuff that’s not very trustworthy and likely to trigger a spam block.

My hope is that mail blocks would be much less likely to happen on the top level server, and if they did you could resolve them much faster.

The lower level servers would be more likely to suffer blocks, but the effects would be limited to (arguably) less important stuff.

@Jeremy,

Completely agree, I get these every couple of months these days. That’s not one email that’s one service blocking my email. For the most part it is minor suppliers here and there but in this case this is a huge one and needs to be resolved quickly.

For the record I have two responses from Trend Micro:

This was the first:

“Hello,

Please re-investigate your network as we still see spam coming from this IP address and contact your ISP and have them to contact us for removal. Thank you.

Kind regards,
Franklynn Uy
Trend Micro Inc.
Spam Investigations Team”

and this was the second (I guess maybe two guys answered my ticket):

“Hello,

Thank you for contacting us about this IP address on RBL. 208.97.132.207 was listed because we received spam from this address and because no action was taken when we reported this to your ISP.

We continue to see spam attempts from this address recently. We need to hear from your ISP prior to removing any part of this listing.

Please secure your network and contact your ISP, and have your ISP contact us for removal.

Kind regards,
John Abello

Spam Investigations Team
Trend Micro Inc.”

It’s incredibly frustrating dealing with this issue. Trend Micro seem to just lie and refuse to giove sane answers for their irrational behaviour.

It’s incredibly frustrating dealing with this issue. Trend Micro seem to just lie and refuse to give sane answers for their irrational behaviour.

Just wondering what legal right do these Trend Micro self-appointed guardians of the internet have to do this and block legal and lawful communications between one person and another like this? One of the worst things is that they do not even inform the recipient that an email to them has been blocked.

Does not the US constitution protect correspondence between individuals under freedom of speech or other clauses?

Why don’t we all start a class action against Trend Micro for their illegal activities and the subsequent costs and loss of reputation they dumb actions have caused all of us?

Any class actions lawyers out there?

@Greg

From their point of view they have a duty to their clients to block spam. If they received (and continue to receive) spam from an IP address they have (as part of the service they provide to their clients) every right to block that IP Address. Dreamhost meanwhile are doing what they can to assist their clients.

You can’t just automatically assume TM are in the wrong.

I LOVE miked ‘s idea of the service levels. I’ve actually repeatedly asked DH for a solution saying I was willing to pay to find a solution.

is it fixed? the warining in the DH panel is gone but my mail dont work yet

@jimbo

still get a bounce back when I try to send an email.

No they don’t. It’s like blocking all emails from a .ru domain on the basis of spam comes from russia. Or, like continually calliong the police because ‘you’ believe your neighbor has broken the law. It’s called harassment. They have legal liability for listing a broad IP address ssrver for what ‘they’ percieve is spam, the result of which is to block tons of valid emails.

The last reposnse from TM is below.

But DH are not doing their job either.:(

It’s time DH moved the servers to different IPs or put in some better relays or technology (which does not use idotic spam lists).

Or, take legal action against TM.

> —–Original Message—–
> From: romina susaya via RT [mailto:rbl@mail-abuse.org]
> Sent: Wednesday, 15 February 2012 1:01 PM
> Subject: [MAPS #229365] (rbl) WWW remove for 208.97.132.66
>
> Hi,
>
> We do not block emails, we simply list IPs that we are seeing recent
> spams coming from. Here is a more recent spam sample for your reference:
>
> ~~~
>
> Received: from [208.97.132.66] by via sendmail with smtp;
> for 1 recipient; Mon, 13 Feb 2012 23:32:47 -0700
> Received: from homiemail-a23.g.dreamhost.com (localhost [127.0.0.1])
> by homiemail-a23.g.dreamhost.com (Postfix) with ESMTP id
> 3633C4B008F
> for <>; Mon, 13 Feb 2012 15:32:47 -0800 (PST)
> DomainKey-Signature:
> DKIM-Signature:
> Received: from Annette (r190-134-45-241.dialup.adsl.anteldata.net.uy
> [190.134.45.
> 241])
> (Authenticated sender: logistock@logikom.com)
> by homiemail-a23.g.dreamhost.com (Postfix) with ESMTPA id
> 6AA024B006D
> for <>; Mon, 13 Feb 2012 15:32:46 -0800 (PST)
> Thread-Topic: Una excelente oportunidad para su Empresa!
> thread-index: Aczqp7Y9bTL0bzGtRomHXDvgCmYsmQ==
> Return-Receipt-To:
> From:
> To:
> Subject: Una excelente oportunidad para su Empresa!
> Date: Tue, 14 Feb 2012 00:32:50 +0100
> Message-ID:
> MIME-Version: 1.0
> Content-Type: multipart/alternative;
> boundary=”—-=_NextPart_000_3588_01CCEAB0.1D093C70″
> X-Mailer: Microsoft CDO for Windows 2000
> Disposition-Notification-To:
> Content-Class: urn:content-classes:message
> Importance: normal
> Priority: normal
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>
> ~~~
>
> Please investigate your network and provide us the following
> information:
>
> 1. What caused the problem that allowed spam to come from this
> machine/server?
> 2. What did you do to fix the problem?
> 3. What are you doing to prevent it from happening again?
> 4. When did you solve the problem?
>
> We need this information to make sure that the problem has been resolved
> prior to removing the IP.
>
>
> Regards,
>
> Spam Investigations Team
> Trend Micro Inc.
>
>
> ——————————————– Managed by Request Tracker

No this is nothing like calling the police because you believe they have broken the law.

This is a service that Trend Micro provides. How they provide it is up to them and it is up to whoever is thinking of using their services to see if it is the service they want.

If I buy my products from an importer, and that importer decides not to import anything from China because of say the melamine in baby powder issue, then it is up to me as the client of that importer whether I want to keep using that provider. Other Chinese manufacturers don’t have any claim against the importer just because they have elected to do it their business that way.

It’s not like TM is forcing everybody in the world to use their anti spam system, my email is still going through to over 90% of the recipients, its just not going through to the ones that have elected to use TM.

I can get around that by sending an email through gmail. It’s not ideal but it solves some of my immediate issues.

“Is anyone else having trouble with Comcast blocking? We’re not getting the mail bounced, it just isn’t being received. It’s like it goes into a big black hole and doesn’t come out. Is it part of this issue or another block that will need to be dealt with separately?”

YES! this is affecting me, too. I’m so glad to see it verified elsewhere, but I am super sad it’s happening to us.

Actually I noticed DH mail servers being listed on RBLs like Spamhaus about 2 months ago.
I always contacted them when emails did bounce, and DH usually filled out the removal form of the RBL.
Nevertheless, they were back on the list withhin hours to days, as the RBL’s honeypot email account kept receiving spam via DH mail servers.
Therefore it seems like DH is unable to identify the spamming customers, and this is probably the same problem with Trend Micro again. Everytime they remove DH, they will receive another spam mail.

@Ji – yes. Exactly. Back on the lists way too fast Some of the sites show the headers and you can identify the sites (where they are using authentication.) But it’s a hydra. I think the problem is there are hundreds or thousands of hacked websites hosted (what did Simon say in his last newsletter? 3 million sites? Yeah more like 10k hacked sites) each of those sites sending spam. stop one and there’s another ready to send. If they don’t stem that tidal wave, they won’t be able to get off the lists. I’ve started paying a 3rd party company to use their outgoing smtp (no plugs here).

I followed the link to mail-abuse.com to request clearing the IP, as many of you have. I received the same 2 responses listed above…and an ironic thing happened: Cactus Spam filter classified them as spam! You gotta be able to laugh in the middle of stressful times (I am copping it from my clients as many cannot send important emails)

As of today, a little before 1AM PST, I still have outgoing mails blocked for
– hotmail.com
– hotmail.fr

from servers 208.113.207.5 and 66.33.216.100..

Unfortunately those mail blocked are sent by a merchant website (invoices and such). I do get the bounces, so I’ve managed, so far to resend them manually,
still sending manually dozens of invoices to customers of my client is NOT in my job description and is quite a drag…

I’m not sure if Hotmail IS a subscriber of the Trend Micro RBL, but in any case the message header read :

Reporting-MTA: dns; crusty.g.dreamhost.com
X-Postfix-Queue-ID: EE9DCBE810
X-Postfix-Sender: rfc822; (name stripped)@jardindorante.fr
Arrival-Date: Wed, 15 Feb 2012 00:56:11 -0800 (PST)

Final-Recipient: rfc822; (name stripped)@hotmail.fr
Original-Recipient: rfc822;(name stripped)@hotmail.fr
Action: failed
Status: 5.0.0
Remote-MTA: dns; mx3.hotmail.com
Diagnostic-Code: smtp; 550 SC-001 (BAY0-MC3-F33) Unfortunately, messages from
208.113.207.5 weren’t sent. Please contact your Internet service provider
since part of their network is on our block list. You can also refer your
provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

The first bounce like that i received from hotmail was dated back on January 31 (yes two weeks ago….)

Guys, I know how hard it is to have an IP removed from blocklists, but two weeks ? That’s pushing it…

posiblemente esto sera eterno.

As a sysmin who’s had to deal with getting servers off of RBLs, I know exactly how hard it can be to get off of some of these lists, and that in some instances, the time that we’re experiencing with DreamHost is not unreasonable.

I can pretty much guarantee that every email anyone has received from Trend’s RBL email address is a form letter, so to those who say “DREAMHOST IS LYING!!!111!111shiftone”, just cool your jets. I know it’s a lot more satisfying to be able to vent at someone, to yell and get mad, but it’s really not worth it. Now, if what Chris said above is true, then it’s likely that DreamHost is having trouble narrowing down the sender of the spam. If that’s the case, then at the 2 week mark, it might be time to call in a consultant to look at their email servers. Not everyone at DreamHost can be an expert at all of the things, and given that mail is probably a low priority service, given the nature of how it works, and that DreamHost encourages every new domain to use Google Apps, I wouldn’t be surprised if DreamHost would love to spin down their email support to just a few key offerings.

Hopefully this will get resolved soon.

Despite of those who would prefer otherwise, I agree with the idea to have DH mass mail such “outage” rather than have cusotmers rely on their DH admin panel visits.

Anyway, I revert back to my gmail account. That works at least.

What is going on today…..my website is down and there is no status post as of today!

Note the source of the spam and the servers originally listed on 2/8/2012 is available at the TM website. This info is in the bounced mails
http://www.mail-abuse.com/cgi-bin/show_listing.cgi?5009169

This means that DH knows the source of the spam and has hopefully terminated those domains/accounts as part of the process.

It’s alarming that this issue wasn’t posted here last Thursday. 6 days late. Just when I thought dreamhost was becoming more transparent with issues as well.

From @sth’s link above, and the way this blog post is worded, I have to wonder if anyone is even regually monitoring abuse@dreamhost.com

@Cody Wilson
Everyone posting here should read your response.

Also, to thems whose never configured a sendmail server (or MSExchange server, etc.) and posting vehement complaints in this thread – you’re showing your complete lack of technical understanding of how blacklisting works.

Lastly: EMail is not now, nor has it ever been, a reliable form of sending messages. If your business model cannot withstand a minimum of 72 hours of email outage:
a) you’re doing it wrong, and/or
b) you should be spending lots more money for electronic mail services, including redundant servers and 24/7 monitoring.

To follow up on your various email block comments: please understand, email blocks happen on a regular basis against all mail service providers. Our Abuse/Security team has a system set up to work with the blocking companies, and most blocks are automatically removed as well. This status post is regarding the block imposed by Trend Micro specifically. If your *recent* bounce message doesn’t list Trend Micro, please submit a ticket through the DreamHost panel, so we can have our Abuse/Security team look into it. Also, if you’re commenting about other emergencies, please also submit a ticket with your details, so we can assist you.

@andrea – I have been posting every bounce I get for the last 4 months. And I’ve been monitoring the blacklist sites myself and DreamHost gets put back on as fast as you get taken off.

Yes, blocks happen to MOST mail service providers, but I have yet to check the:

http://whatismyipaddress.com/blacklist-check

site and NOT find my main DreamHost IP servers listed somewhere. (Go ahead, look at any of these Dreamhost IPs:

208.97.132.66 or 208.113.200.5 or 208.113.207.5 or 208.97.132.207

ALL are listed on at least 3 of those sites. By comparison, I have been looking at other mail services (again, I’m not advertising for them so I won’t post names or IP addresses) and so far NONE have been listed on ANY of those sites! That is: Dreamhost is notably WORSE on this account. I know it is because of size, but like someone else posted: why not offer a “clean” IP address to people who are willing to pay.

Still can’t send to statefarm.com due to Trend

all of my small brochure site clients emails are down! down!

will they ever fix this?

@jimbo… you might want to open a different support ticket for that one. This issue doesn’t stop your email from working, I can send email to 90%+ of people I need to send email to. The issue here is that some emails are being bounced back because Trend Micro has blocked the IP address of all Dream Host shared servers.

We sent out the email Sunday night and I just got 28 bounced back from Comcast. They aren’t coming in all at once but they are coming. I know this issue is dealing with Trends RBL. The IP seems to be blacklisted in other place also such as spam.dnsbl.sorbs.net, bl.emailbasura.org, blacklist.spambag.org
Does anyone know if these all part of this same issue with Trend or do they need to be submitted to DH separately?

@Rebecca – I think you might want to submit a different ticket for that too based on Andrea’s earlier comment:

“andrea Says:
To follow up on your various email block comments: please understand, email blocks happen on a regular basis against all mail service providers. Our Abuse/Security team has a system set up to work with the blocking companies, and most blocks are automatically removed as well. This status post is regarding the block imposed by Trend Micro specifically. If your *recent* bounce message doesn’t list Trend Micro, please submit a ticket through the DreamHost panel, so we can have our Abuse/Security team look into it. Also, if you’re commenting about other emergencies, please also submit a ticket with your details, so we can assist you.”

Hi Dreamhost,

Is there any update on this Trend Micro issue? Are you guys in regular contact still?

Thanks,
Jo

February 16th, 2012 at 4:39 amMarcelo Oliveira Says:

problem continues :

This is the mail system at host crusty.g.dreamhost.com.

I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.

If you are a current customer of DreamHost, please contact our
technical support team here

https://panel.dreamhost.com/support

If you are not a customer please use our contact form at.

http://dreamhost.com/contact

If you do so, please include this email in your support ticket. You can
delete your own text from the attached returned message.

Dreamhost Email Support

: host mx3.hotmail.com[65.54.188.126] said: 550
SC-001 (BAY0-MC4-F45) Unfortunately, messages from 208.113.207.5 weren’t
sent. Please contact your Internet service provider since part of their
network is on our block list. You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL
FROM command)

Final-Recipient: rfc822; @hotmail.com.br
Original-Recipient: rfc822; @hotmail.com.br
Action: failed
Status: 5.0.0
Remote-MTA: dns; mx3.hotmail.com
Diagnostic-Code: smtp; 550 SC-001 (BAY0-MC4-F45) Unfortunately, messages from
208.113.207.5 weren’t sent. Please contact your Internet service provider
since part of their network is on our block list. You can also refer your
provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

The issued is STILL NOT RESOLVED!!! Day three since it was brought to your attention! GET IT FIXED!

Hello, I would also like an update on the issue,
The majority of users on my site are hotmail users, and almost every single email to a hotmail.co.uk/com/live etc are bouncing back.

It was mentioned above that someone might be interested in a class action…
Im wondering if it could also be argued that Trend are falsely accusing anyone who sends an email/notification/newsletter etc as spammers.
Especially as its for their own commercial gain at the expense of genuine website owners. Could these be deemed as something along the lines of anti-competitive behaviour, although they aren’t, as far as I am aware, in competition with dreamhost?

I’m suprised that this isn’t something that this hasn’t caught the attention of the more mainstream media. I’m sure Trend Micro would soon improve the way the operate given the right type of publicity….

Hmm just recieved an email from a friends hotmail account,
Interestingly it points to a gif file hosted on phamsp.org – a wordpress installation apparently hosted by dreamhost :/

Actually I stand corrected, it points at a php file that has a variable tagged onto the end to make it look like a .gif file…
It redirects to a http://profitsathomeguide.net a site registered in russia….

When I contacted for the blacklisted IP [66.33.216.122‏] to Trend Micro RBL, they said:

This IP is listed because we received spam from this address and because no action was taken, when we reported this to your ISP. Any removal request of this IP must come from them or any authoritative administrator of this IP, so you may just invite your ISP to contact us at rbl@mail-abuse.com to send their removal request.

Situation is still same. No action from DH and no messages couldn’t send from blacklisted IP and no action to fix it until 2 days.

Pretty mess..

Dreamhost, we need an update on this please. Haven’t heard from you in a while. The natives are beyond restless.

OK, I sent in an additional Trouble ticket for the Comcast issue. By this morning I had received 118 bounced emails for Comcast customers. One of the things that bothers me is that even after they get this fixed it’s not possible to resend to just these customers. If we resend the announcement everyone will get it and that would annoy the majority of our customers.

According to mxtoolbox.com, we’ve been blacklisted on:

BACKSCATTERER
CASA-CBL
CASA-CBL+
NIXSPAM
REDHAWK
SORBS-SPAM
SPAMCANNIBAL
UCEPROTECTL1

PLEASE GET US OFF OF THESE.

Can you guys not just stick the servers onto a different IP address?

I don’t understand why this is taking so long – its been several days since DH was made aware of this.

We’re listed on
SORBS-DUHL
Spanhaus-ZEN

I Agree with Fatih – pretty mess up…. From Andrea’s posts (Andrea from DH) it almost sounds as though people are pointing fingers…instead of pointing. keep us informed as to the progress, and get resolved. Several days is not acceptable

come on DH, respond the mails a was send.

I agree with Stacey – I know that it is mostly outside of DreamHosts control, but I am DH can more than afford a lawyer to start proceedings against Trend?

It is inevitable that DH are going to loose customers to other hosts just so that they can send emails. And no doubt, people who use DH to run a business, send invoices, delivery notifications etc are being cost substantial amounts, but not realise why emails are bouncing..

I am sure Trend and other such companies are liable for damages to DH and its customers?

This is what TREND told me today, please do something.

Hi,

Thank you for your email. In any case, this range of IPs is owned by Dream Host. Since it is their space and they are ultimately responsible for it, any request for removal must come from them. If you happen to speak to your ISP, please invite them to contact us at this address, rbl@mail-abuse.org, so that we can resolve this issue. Thanks again.

Kind regards,

Juan Miguel N. Bartolome
Spam Investigator Engineer
Trend Micro Inc.

It is my opinion that because we are not the ISP, when we contact Trend or Comcast or anyone blocking us, we get the low level tech with a canned response. It doesn’t really indicate that DH isn’t working on this. What I don’t understand is why the “spam” being received and identified by Trend, can’t be traced to the offending account(s).

Note that most of those automatic responses from Trend Micro are actually “out of context”. Note they are all identical – it is an autoresponder.

For all us legitimate businesses who are trying to get our mail out, this is a real problem.

The real dilemma is how can dreamhost stay OFF these lists. Any of the spam-trapped messages I’ve seen are clearly spam and, assuming the best of my fellow DH members, these must be from compromised (hacked) accounts and/or websites. There is lots of evidence that at least one route in is by wordpress vulnerabilities which are being used to repeatedly open up sites to world-writable state, infect them with hidden viruses and then send spam (I worked directly with Dreamhost staff to reverse-engineer one such virus from a shared system). Unless DreamHost finds a way to stop that, we’ll keep being blacklisted.

@leviatan – mine are worse.

7 listings for 208.113.200.5 (authenticated send from caibbdcaaaaf.dreamhost.com )
2 listings for 208.113.207.5 (unauthenticated send via PHP’s mail() command from apache2-echo.dart.dreamhost.com )

http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3a208.113.200.5
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3a208.113.207.5

(well, my comment is awaiting moderation… so, without links, I found 7 (SEVEN) additional black-lists for the IP address my users send mail through. SEVEN!)

From the dialogue posted already, it would seem appropriate for DH to take some additional actions through Professional Associations and groups concerned with these issues, specifically identifying Trend Micro as being irresponsible in its method of “group punishment.” If DH is to blame for a failure to adequately monitor SPAM, this will become obvious from push back from TM and other colleagues. If TM is responsible, the pressure from peers and the competitive pressure from other blacklist providers should solve that problem.
If DH is unwilling to take such actions and publicize the results we should all move to another ISP (and likely will).
Another useful tool in response to TM’s intransigence would be to publish the email, phone (including home phones) and physical points of contact with the key TM officers and board members. A rising tide of email and eventually of late night calls at home will have its effect.
Lastly, if this is not taken seriously, I am afraid we will find no shortage of pressures to have government involvement whether that be through congressional investigations, executive branch monitoring or litigation in the courts. Can’t we avoid this by getting some serious action among peers?

We appear to be back on a blocklist again =/

meanwhile checkout payklips.com xD

yeah i’m still experiencing this issue. i can’t even login to my web panel now because of it. i have it setup to notify me about logins from new IP addresses. i recently changed addresses and now can’t access my account because i’m not receiving the permissions email. i used to think it was a good idea to have this security feature, now i’m thinking it’s more trouble than it’s worth?

is there any way around this for me? i really need access to my web panel soon.

Still seeing messages bouncing here, despite DH telling us this is fixed. Have reopened my support ticket.

It was fixed for all of a day or two – i had no bounces at all but this weekend its kicked off big time

The Comcast email issue *might* be separate from this issue, see my post here: http://discussion.dreamhost.com/thread-134276.html

PHP mail from my site is not getting to any AOL, Comcast and now Yahoo! accounts. We’re coming up on the 24 hour mark since the last update and it’s still not working.

I still can’t sent any mails to hotmail addresses. This is getting really annoying. I can’t “start” putting my website online (for the first time), because a lot of my users wouldn’t be able to register and receive the activation email.

PLEASE sort this as soon as possible. I am waiting for several day’s now to get this fixed.

now our messages are also blocked by UCEPROTECT. some, not all messages are being blocked by both trend and uceprotect because they are coming from dreamhost servers. this is the first time since we signed on w/ them that we have had these issues. first hardware capacity, then software/backup/restores, then ddos, now blocking due to spam, what next?

This issue is still ongoing with mail coming from my discussion lists but private email from me to anyone on AOL is going through just fine. This is inline with the latest update posted for Feb 20th.

Still getting bounces:
host mx3.hotmail.com[65.54.188.94] said: 550 SC-001
(BAY0-MC2-F38) Unfortunately, messages from 208.113.207.5 weren’t sent.
Please contact your Internet service provider since part of their network
is on our block list. You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL
FROM command)

Yahoo seems to be working now, but AOL and Comcast still are not receiving php mail.

AOL still not receiving email. MSN & Hotmail bouncing still. Per Andrea’s request, 2/23 post, I’ve opened a support ticket.

well, filed a ticket and no reply yet =/

it’s been over TEN DAYS since the discussion lists broke. PLEASE, dreamhost (and TrendMicro), please fix this!

2/27/12

Our discussion list at lists.confluencechorus.org is STILL not working and you have not posted an update since saying you are “well aware” of the problem.

I’ve been a Dreamhost customer for almost 7 years and this is by far the WORST service from you. You have not provided any updates on the status page for almost TWO WEEKS. Telling me you are “well aware” of the problem is insulting and condescending. You need to at have enough respect for me and others who suffer from this problem to report IN DETAIL exactly what you are doing, when you expect the problem to be resolved, and any suggestions you have from your customers re how we can be of assistance.

I just submitted another ticket since some of our php email is not being delivered. I can verify the email is not delivered to Comcast customers. This includes new accounts, order confirmation, order updates, password resets. so far the mail hasn’t been bounced from Comcast but when this happened on 2/12 it took four days before we received the announcement list bounces. I also think that when this happened last July we never received any undelivered. The email just never arrived. Customers can’t order without logging in and can’t log in without a password. Difficult to do business if passwords aren’t delivered and customers can’t log in. Who do we bill for lost sales?

@Rebecca – Comcast blocks are not related to this status message, we’re investigating them separately. I have your open ticket and will get back to you as soon as I have some news.
@Bob Winget – Unfortunately, the discussion list server you’re on is still blocked by Trend Micro, despite our ongoing efforts. We also have been updating the status post you’re commenting on – what are you referring to when you mention no updates for two weeks? Let me know, so I can look into it for you. If you wish, and depending on the status tomorrow, I can request a server move for your discussion list. Please note, however, that it may take a day for the new DNS records to propagate through all your subscribers’ ISPs. I will also email you this option.
@Ji – The discussion list move option is also open to you. Please submit a support ticket if you haven’t yet (I’m unable to check without your details) if you would like me to follow up on that for you too.
@GJ – Could you let me know your domain or webid, so I can follow up on your ticket?
@Geoff – those are separate blocks from this case, and as far as we could tell at the time we checked them, they were random. Please give me your domain name or webid, so I can follow up on your support ticket.

Thanks for responding, Andrea! If have to move my discussion list, I’m going to move it to MailChimp. I’d prefer to have everything under one roof, but this has been ridiculous!

@ji – the move I offered would be in house: our other discussion list server isn’t blocked by Trend Micro.

All blacklists are run by abusers.

Does anyone else notice that the big email providers like Gmail rely on more accurate methods?

There’s a reason…

@Andrea – scrimmagesonline.com

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver158 being moved to new hardware and restoring from backups ( RESOLVED )
6:41 am

Web VPS host homie-vserver158 being moved to new hardware and restoring from backups ( RESOLVED )

Posted (February 13th, 2012 at 6:41 am PST) by Justin@DH

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver158′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver158’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:59pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Update Feb 22nd, 2:00am PST: It would appear that this restoration is now complete! We will wait until morning to mark this resolved, so that our Admins can have a look to make sure everything is complete. Thanks for your patience with this process. Please Contact Support if you notice any issues with your VPS on this host server! -Patrick

Update Feb 21st, 8:08pm PST: All VPS guests are running and the server itself remains stable. We’re just waiting on the one last VPS guest to finish restoring. If you notice any issues with your VPS, please contact support. We’ll continue to post updates as the restore process completes. – Seohee

Update Feb 21st, 1:30pm PST: All VPS guests are running and the server itself remains stable. We’re just waiting on the last few VPS guests to finish restoring. If you notice any issues with your VPS, please contact support. We’ll continue to post updates as the restore process completes. – Oscar

Update Feb 21st, 6:00am PST: All VPS guests are running and the server itself remains stable. We’re just waiting on the last few VPS guests to finish restoring. If you notice any issues with your VPS, please contact support. We’ll continue to post updates as the restore process completes. – Chalise

Update Feb 20th, 7:47pm PST: Only a few VPS guests remain before this restore is complete! All VPS guests are running and the server itself is stable. If you do notice any issues with your VPS, please be sure to contact support. We’ll continue to post updates as the restore process completes. – Dan J

Update Feb 20th, 5:25am PST: Only a few VPS guests remain before this restore is complete! All VPS guests are running and the server itself is stable. If you do notice any issues with your VPS, please be sure to contact support. We’ll continue to post updates as the restore process completes. – Chalise

Update Feb 19th, 7:00pm PST: We are still getting very close to having this completed. The server remains stable and VPS guests are all running, but if you notice any issues with your service, please contact support. This post will be updated while the restore process finishes up. – Seohee

Update Feb 19th, 8:45am PST: We are still getting very close to having this completed. The server remains stable and VPS guests are all running, but if you notice any issues with your service, please contact support. This post will be updated while the restore process finishes up. -Justin

Update Feb 18th, 11:25pm PST: We are nearly there, just a couple more guests to go before the process has completed. The server remains stable and VPS guests are all running, but if you notice any issues with your service, please contact support. This post will be updated while the restore process finishes up. -JJ G

Update Feb 18th, 12:35pm PST: The restore process is almost done! There are a few more guests that need to finish restoring. We’ll keep updating this post as it progresses. If you notice any issues with your service, please contact support and we will be happy to help! -Oscar

Update Feb 18th, 6:44am PST: The restoration of data to this new server is not encountering any difficulty at this time. If you notice any issues with your service, please contact support and we will be happy to help! -Patrick

Update Feb 18th, 1:52am PST: This restore is moving right along still! All VPSs are reporting online and we are on the last batch of restores before this is complete! If you notice any issues with your service, please contact support and we will be happy to help! -Patrick

Update Feb 17th, 9:45pm PST: The restore process is running well. We will continue to update this post as the process moves along. Your sites should be accessible as the restore is happening. If you see problems with your service, please let us know via our Support System. Thanks, once again, for your patience! –Dan J

Update Feb 17th, 11:50am PST: Our system administration team has fixed the issue with the backup drives and the restoration is now continuing without any other issues! All VPS guests are reporting online as well. If you notice any missing or old data, or if you are experiencing any issues, please contact supportOscar

Update Feb 17th, 4:00am PST: We’re currently experiencing a few issues with the backup brick that’s restoring data to the remaining guests. Our admin team is currently looking into this and we will post another update shortly. Thanks for your patience! – Chalise

Update Feb 16th, 9:56pm PST: This restoration is continuing with no issues at this time! All VPS guests are reporting online. If you notice any missing or old data, or if you are experiencing any issues, please contact support! - Seohee

Update Feb 16th, 5:20am PST: This restoration is continuing with no issues at this time! All VPS guests are reporting online. If you notice any missing or old data, or if you are experiencing any issues, please contact support! -Patrick

Update Feb 16th, 2:00am PST: All VPS guest machines are up and running using NFS mounted backups. The restoration process continues to execute. If you’re missing data, please contact support for assistance. More updates to come as the restore progresses. We appreciate your patience with this matter. – JJ G

Update Feb 15th, 7:26pm PST: All VPS guest machines are up and running on the new server, using NFS mounted backups. Data is continuing to be restored to each guest server one by one. If you’re missing data, please contact support for assistance. More updates to come as the restore progresses. Thanks once again for your patience! – Seohee

Update Feb 15th, 1:15pm PST: All VPS guest machines are up and running on the new server, using NFS mounted backups. Data is being restored to each guest server one by one, and is continuing to progress. If you’re missing data, please contact support for assistance. More updates to come as the restore progresses. Thanks once again for your patience! – Oscar

Update Feb 15th, 3:20am PST: All VPS guest machines are up and running on the new server, using NFS mounted backups. Data is being restored to each guest server one by one, and is continuing to progress. If you’re missing data, please contact support for assistance. More updates to come as the restore progresses. Thanks once again for your patience! – Chalise

Update Feb 14th, 9:34pm PST: All VPS guest machines are now up and running on the new server, using NFS mounted backups. The data move to new hardware is progressing, and it’s going through all guest servers one by one. If you’re missing data, then it’s not on the backup server, so please contact support for assistance. We will continue to update this post  as the restore progresses. Thank you for your patience. -Dan J

Update Feb 14th, 3:15pm PST: All VPS guest machines are up and running off mounted backups. The restore is still running smoothly as well. If you’re missing data, please contact support for assistance. We will continue to update this post  as the restore progresses. Thank you for your patience. -Oscar

Update Feb 14th, 5:31am PST: The restore is still on track and all VPS guests are reporting online at this time! If you notice any issues, or are missing any data, please contact support for help! -Patrick

Update Feb 14th, 1:56am PST: First, Happy Valentine’s Day! This server restore is running smoothly and all VPS guests are reporting online! If you are experiencing any downtime, or are missing any data, please contact support for help! -Patrick

Update Feb 13, 9:31pm PST: All VPS guest machines are now up and running on the new server, using NFS mounted backups. The data move to new hardware is progressing, and it’s going through all guest servers one by one. If you’re missing data, then it’s not on the backup server, so please contact support for assistance. We will continue to update this post  as the restore progresses. Thank you for your patience. -Dan J

Update Feb 13, 8:36pm PST: The backup server for homie-vserver158 became unresponsive earlier and that stopped the restore process. Our admins are were able to get the backup server running again and the restore process is underway again. There are currently a few VPS that are stopped but the admin team are working to get those running again using the backup server files. We will continue to update this post  as the restore progresses. Thank you for your patience. -Dan J

Update Feb 13, 7:17pm PST: The backup server for homie-vserver158 became unresponsive earlier and that stopped the restore process. Our admins are were able to get the backup server running again and the restore process is underway again. There are currently a few VPS that are stopped but the admin team are working to get those running again using the backup server files. We will continue to update this post  as the restore progresses. Thank you for your patience. -Dan J

Update Feb 13, 12:50pm PST: All VPS guest machines are now up and running on the new server, using NFS mounted backups. The data move to new hardware has also started, and it’s going through all guest servers one by one. If you’re missing data, then it’s not on the backup server, so please contact support for assistance. We will continue to update this post  as the restore progresses. Thank you for your patience. -Brandon

Update Feb 13, 9:30am PST: The replacement hardware required additional preparation to get it ready for to replace this vps host machine. It is now up and ready to go and the VPS creation process can begin very shortly. Each VPS will need to be built new, mount the backup server for its /home and then the will be up and working. The restore process can then work in the background over the next few days to complete the restore. When we have an update or all the VPS have been created, we will update the post. – Justin

Update Feb 13, 6:00am PST: Replacement hardware is already setup and running in the datacenter, and our admins have been called to start the process of mounting the backups to the new hardware, creating the VPS and starting the backup process. When we have further info on how long this process is expected to take, we will let you know. Our apologies for this unplanned downtime. -Justin

This entry was posted on Monday, February 13th, 2012 at 6:41 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

86 Responses to “Web VPS host homie-vserver158 being moved to new hardware and restoring from backups ( RESOLVED )”

Affected VPS:
ps23363 ps23387 ps23460 ps23667 ps23826 ps23862 ps24475 ps24476 ps24548 ps24605 ps24669 ps24739 ps24988 ps25019 ps25032 ps25248 ps25321 ps25398 ps25416 ps25496 ps25602 ps25671 ps25722 ps25989 ps26137 ps26242 ps26626 ps26845 ps27433 ps27554 ps28181 ps28183 ps29385 ps29424 ps29924 ps37690 ps42470 ps42484 ps44099 ps44106 ps76226

Looked like it popped up for a bit, then it went back down unfortunately. Hope this gets resolved soon!

@Tony: We are doing everything we can to get replacement hardware up and running and VPS built. Sorry that this affects you.

Any status? ETA on fix?

We are not on the list, but we have VPS down.
ps20395

Ya, this is about to push on 4 hours since I first noticed it… this is very unacceptable.

@Steve: I just updated the post. I understand the process seems slow, unfortunately a replacement of vps hardware is more involved due to the many individual vps, operating systems, special kernel. We are really doing all we can as fast as we can. The VPS will start building now, meaning they will slowly start coming back up.

@TK: You are not on his host machine, and your host machine seems fine. I see your VPS has a 17 minute uptime, I believe your issue should be fixed.

@Tony: I am aware that this is taking longer than expected, we are doing all we can.

I am unable to make changes on my website.

No problem.. at least you guys keep us updated.

From reading your update, are you saying that their is a chance it could be DAYS before our sites are back up?

@Tony: Nope, not days, I am hoping it won’t be more than a few hours to have all VPS back up and running. The VPS run off of the backup server for their data, while in the background the data is also being copied to the local storage of the new webserver.

@Debbie: You are not affected by this issue. Please submit a ticket with a step by step process with all needed info to reproduce your issue and we would be happy to help.

Awesome, thanks for the responses Justin.

Who do we contact to get our account credited for the downtime? Your guarantee says for every hour we get a credit of 1 day.

ours has been down since 6am this morning. I want credit. This has happened numerous times and I have never received credit for any of them.

@Terri and Tony: Please open a support ticket and when the issue is resolved we will take care of that.

Already did this morning. Thanks though. I would like to know how the credit thing works though. Thanks for replying so fast!

@Terri: On this page, the bottom is a tab for satisfaction guarantee. http://dreamhost.com/web-hosting/

Sites somewhat back in and out. Long ping times. WordPress posts are stripped of images (in the source code when viewing the page). Some WP plugins dead in the water. Will this stabilize and when please?

ya, getting problems with my word press install also, getting errors on a lot of plugins, jquery wasn’t being loaded… seems ok now.

It’s being really finicky at the moment.

my email is not working for one of my addresses (I can’t receive or send), would that be due to this issue?

My images are stored in wp-content which is under https. A call to an image or anything under https is throwing a 404, that is when I can connect at all. The images do exist on the server. I suspect that is what is in play with some of the plugins too. The backup storage needs to be mounted in such a way that https works…. Please :)

Any ideas on when this is going to be fixed? Our ruby rails could not be started error message is up but nothing working.

Images aren’t loading on at least a couple of our WordPress sites. I submitted a support ticket.

fuck! this is ridiculous. I’m so fed up with you guys.

Going on 5:00pm now… a whole day is lost… site is slower then molasses and doesn’t load half the time still. Still missing files… plugins still broke. Now my site doesn’t even load, its just timing out.

Can you please update us on how long its going to take? Even an approximation on timing will help. We will have to plan accordingly.

This is totally unacceptable. No emails for a whole day!!!! We are not happy with your service.

Can anyone tell me how long does it usually take for support to respond to a ticket? I have 2 tickets in and it’s nearly 22hrs and not call yet.

Any suggestions?
Barbara

Ok so now I see a text response that my issue has been turned over to security but I don’t know what the hell that means.

I was under the impression when you elect to pay for a call back that they actually call. Any suggestions how I can contact them to find things out.

Barbara

My site went back online around mid day to early afternoon today. It’s is now back offline. What’s going on? Did the server go down again?

DH can you please communicate? The whole day is shot.

We have not been able to receive emails all day, although the emails we sent have gone out. When will this be fixed????

Update please. What is the status now? Is anybody working on the server?

My VPS came back for a bit, it though just kicked out again. It would appear I am missing 7 days of files. I am making the assumption that the backup mounted is from 2/7/12. The question is when homie-vserver158 comes back again and everything is restored to local storage will I have a data loss situation at the end of the day?

Will Dreamhost be restoring from a week old backup? Please advise, I do not want to go in and start replacing the missing items if they will be restored. Thanks.

Thank you so much. I can understand how much tedious it is to get the server back up and running after a hardware crash. Will check the data and get back. :)

All my domains have been out of emails for almost 24 hours. I am getting angry phone calls now. I know how hard it is to restore service, I am starting to fear that we will get bounced emails now.

homiesub4 , the email server for my domains, is not back on. Any news? in 15 minutes it will be 9am EST…

What is going on. We are losing a TON of business because of this. It needs to be fixed NOW!!! We get numerous leads off of our site and it being down is detrimental to our business. This is UNACCEPTABLE!!!

My domains are up and running and so are the databases. But the access is very slow.

I am also missing files from the past few days.

Someone needs to update us on this. I have even asked for a call back and no one has called me!

We are getting e-mails back saying they are Undeliverable because Trend Micro has detected us a spammers. Trend Micro is apparently a big outfit. They say the dreamhost server has been used as spammer, so we have been black ball’d. Is this due to switching servers? Help please. Support is slow to reply. You seem to reply better here. Thank you.

@Dana & @Chris: Please contact support for further information about mail blocks.

@Terri: I am working on your servers myself right now. I am getting closer to having it completed and working normally.

@Tony: The backups will have some delay from what your live site had and what the backups have. Please contact support if you haven’t, as we should be able to get files from the old server if it is in a state to do so.

@Ritu: The access unfortunately will be slow. If you want to contact support, requesting Justin, I can see what I can do about it.

Trend Micro is blocking all emails originating from our main domain. Based on the lookup of their RBL database, a bunch of Dreamhost servers are black holed. I have already submitted a support request and would like to know how quickly this will get resolved.

@Justin: I have contacted support 6 times since this began happening yesterday afternoon. All of a sudden, we have been blacklisted by at 6 major anti-spam. People we could send to as recent as yesterday morning began bouncing yesterday afternoon. This has to have something to do with your switching servers. Either get us un-blacklisted or we, too, have had enough of your service.

Thank you.

Sites are back up but blog images still aren’t loading. Still no word from support. And from the comments above it looks like I’ll have to check with clients to make sure their latest blog posts aren’t missing?!

@Terri: Your service should be restored.

@Dana: Our abuse team has been in contact with the parties necessary since Friday to get the mail block issue resolved, we are hoping for a resolution very soon. It has nothing to do with this move to new hardware, we use load balanced mail servers that are separate from the webservers for outgoing mail. It is not possible to work around this issue at the time. Tickets that have been submitted about this issue should be getting email updates about any progress or news.

@Jesse: I can’t find a ticket open with the name “Jesse”, if you provide a domain name I can try to find your ticket and get to it myself.

@Justin: almost all of my sites are now back up, though load time is slow. SSH is up as well though access is slow. But my Ruby on Rails site is *not* up.

ugh, now my website is down again. and I am still getting errors with a wordpress plugin ever since the transfer.

@Tony: I am working on your site right now. I am getting it so it is on the local disk, will not be as slow and will recover the missing files for you from the old server. Please standby.

@Bob: I will be looking at your tickets and working on your VPS next.

Sites down again.

I replaced the critical missing files myself. still, there are others I would like back.

When the sites were up they were painfully slow.

I have put in numerous tickets since yesterday. Haven’t gotten a response since yesterday AM when this first started. I would like my files. I would like my sites to function as they usually do. With the 1/29 outage, I estimate I have been dead in the water 40 plus hours over the last 3 weeks.

Anything you can do to move this along would be greatly appreciated.

@Steve: I just replied to you, your files should be restored, and your sites working. The slowness unfortunately I can only do so much, until the restore gets closer to completed.

hey steve,

just an FYI your site is working fine on my end. Albeit a bit slow, everything is loading for me. Just to ease your pain a little bit i hope.

Any ETA on my site justin?

@Tony: 3.1 of 5.3 GB copied for petersenhealthcare.net, hoping it will be no more than another 20 mins for that site, then working on the rest which are much smaller.

Thanks :) Lotttts of pictures up there.

Not back online. This is absolutely absurd. I ordered a new VPS and tried to point one of our subdomains over and it’s still not working. Please fix this NOW!

Site is back up, running much faster. Thanks Justin I will report back with any problems.

Thanks Justin, Thanks Tony.

The restore of the missing files seems to have overwritten a bunch of stuff I added this AM. Not really a big deal I can replace that stuff easily, so no need to do anything.

Load times have improved exponentially.

@Justin I did ask for your support but nobody got back to me. Will take a while before you can contact us back? The websites and databases are really slow.

@Steve: The changes you made this AM are still on the backup server, if you provide a detailed list of files I can get them for you.

@Ritu: I’ll see if I can find your ticket and work on your issue soon.

@Justin – this is what I got reply from support (#4748733):

Thank you for contacting DreamHost technical support. As the restore is still in progress we are aware that there is slow load time and timeout issue from the restore taking place. Once we have completed the restore if you are still experiencing these issues we will be able to investigate these issue further. Please keep an eye on the status updates as they become available and if we have marked the issue as resolved and you are still experiencing these issues contact us back.
http://www.dreamhoststatus.com/2012/02/13/web-vps-host-homie-vserver158-being-moved-to-new-hardware-and-restoring-from-backups/
We are truly sorry for any delays you have experienced and look forward to this being resolved as quickly as possible.
Thanks!Greg

I’ve been waiting for support to resolve my MySQL issue for awhile now. Support chat is currently offline and I attempted to restart the service several times.

Restart MySQL 2012-02-14 10:30:29 Failed
Restart MySQL 2012-02-14 07:49:47 Failed
Restart MySQL 2012-02-14 06:51:00 Failed

Thanks,

@Customer: I fixed your re-enable issue, you will have a reply shortly from one of our techs.

@Ritu: I’ll work on your issue now.

@Ritu: Replied to you, you should have some improvement now.

No email, no website. My coproducer’s website is out here as well (in Sweden) but his is working on the west coast, mine is not.

My panel says I have nothing on the site either. Sad to pay and be left behind.

Our site’s currently unresponsive.

@Justin Thanks! Everything Great :)

Our site is still down! What is going on?

Folks seem to be getting better response here than I have been getting from support tickets for the past two days (we’ve been down for two days now – my people are just sitting on their hands waiting).

Justin, could you please check on gonola.com for me and see if I can get some movement there? Thanks!!!

@Justin – the speed of other sites and databases is doing great now. Thanks.

Site has gone down 40 times in the past two days for a total of nearly 18 hours. This is ridiculous.

@Justin – Is there any update on ps25602 (gondola.com)? We are entering the 3rd day of being offline.

@Nathan: I found your ticket and have restored your VPS to local disk, it should be working better now. I wrote you with some tips on your backups as well.

@Curtis: I’ll take a look and see what I can do. It appears your VPS is using around 400+GB of data, which is not a small feat to restore over the network.

Justin,

We’ve lost some revisions from our subversion virtual server. We need those restored as soon as possible. Can you give us an estimate as to how long this will take to restore?

We’re still experiencing downtime here and there. When the site is up and actually working, it’s impossibly slow. Any updates?

@Dan: The restore from backups will very likely not have all the revisions that the live server had, due to the lag in backups. I can rsync over from the old server for you though, please submit a ticket with a subject such as attention Justin or need data restored from old hardware, and we’ll get them rsync’ed back up.

@Dan: I emailed you back, asking for specifics on your missing data. I don’t see any svn repos on your VPS, at least ones setup from our control panel.

@Matt: I’ll take a look at your VPS now.

We have all of our revisions back now. Thanks!

Justin I just lost a whole bunch of recently uploaded images…

@Steve: I just rsync’ed your VPS again from the old webserver, which copied over a few dozen uploaded images for a wordpress install. Give it a look now, if you still see issues please let me know an example of a missing image in a support ticket.

I am also missing about a weeks worth of uploads again. Anything after 2/7 – 2/18 it seems like. I submitted a support ticket on the issue.

@Tony: I replied to your ticket. Need more information, since no reverting of the data has happened to my knowledge and no specific files having issues or missing was provided.

Justin, I apologize, I just noticed your @ response. My colleague Dale and I have both been emailing support, and the account is in his name. One of the affected sites is stulagu.com aka studiolaguna.com
We have missing data and I am having a terrible time getting someone from support to take a serious look at it. Can you help?

@Jesse – The previous responses were correct in that our backups definitely didn’t contain that data, however, I was able to locate it on the old hardware. Replying to your open ticket now!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver206 being moved to new hardware and restoring from backups (RESOLVED)
6:24 am

Web VPS host homie-vserver206 being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 9th, 2012 at 6:24 am PST) by Justin@DH

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver206′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver206’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 14th, 12:00pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Justin

Update Feb 14th, 5:32am PST: The restore is still on track and all VPS guests are reporting online at this time! If you notice any issues, or are missing any data, please contact support for help! -Patrick

Update Feb 14th, 1:58am PST: First, Happy Valentine’s Day! This server restore is running smoothly and all VPS guests are reporting online! If you are experiencing any downtime, or are missing any data, please contact support for help! -Patrick

Update Feb 13th, 8:32pm PST: The restore process is continuing to run smoothly and without any issues. There may be slight slowness if your VPS is still being hosted by the nfs mount instead of local disk, but as the restore continues, there will be fewer and fewer VPS being affected by this. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. We will continue to provide you with updates. – Dan J

Update Feb 13th, 3:50pm PST: The restore process is continuing to run smoothly and without any issues. There may be slight slowness if your VPS is still being hosted by the nfs mount instead of local disk, but as the restore continues, there will be fewer and fewer VPS being affected by this. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. We will continue to provide you with updates. – Brian S.

Update Feb 13th, 12:20am PST: The restore process is continuing to run smoothly and without any issues. There may be slight slowness if your VPS is still being hosted by the nfs mount instead of local disk, but as the restore continues, there will be fewer and fewer VPS being affected by this. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. We will continue to provide you with updates. – Chalise.

Update Feb 12th, 3:43pm PST: The restore continues to run well. There can be slight slowness if your VPS is still being hosted by the nfs mount instead of local disk, but as the restore continues, there will be fewer and fewer VPS being affected by this. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. We will continue to provide you with updates. -Seohee

Update Feb 12th, 9:00am PST: The restore continues to run well. There can be slight slowness if your VPS is still being hosted by the nfs mount instead of local disk, but as the restore continues, there will be fewer and fewer VPS being affected by this. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. -Justin

Update Feb 12th, 12:00am PST: The restore continues to run, due to this you may notice your VPS or sites responding slowly. The restore process and the machine will continue to be monitored. Please contact support with any questions or issues. -JJ G

Update Feb 11th, 5:40pm PST: The restore is still running and is about halfway done. While the restore is running you may notice your VPS or sites respond slowly. This is due to the ongoing restore, but our admin team has been working to get your VPS’s stable. Response times are looking much better now. Our admin team is continuing to monitor the restore and make sure it progresses smoothly. Please contact support with any questions! -Oscar

Update Feb 11th, 11:00am PST: All but one VPS guest on this host are up and running while this restore continues! We are investigating the issue with the one VPS now! Please contact support with any questions! -Oscar

Update Feb 11th, 6:54am PST: All but one VPS guest on this host are up and running while this restore continues! We are investigating the issue with the one VPS now! Please contact support with any questions! -Patrick

Update Feb 11th, 2:57am PST: Our Admin team continue to work on a couple of VPS guests on this server, all while the restore continues! Please feel free to contact support with any questions, or concerns! -Patrick

Update Feb 10th, 9:38pm PST: Our admin team are reporting that the data volume on homie-vserver206 is showing errors. The admins are currently prepping a new machine for homie-vserver206. This may take a bit. Once the machine has been prepped they will mount the backup server again and continue with the restore. We’ll post more updates here as soon as we get them from the admin team. -Dan J

Update Feb 10th, 8:59pm PST: The server appears to have become unresponsive. Our admin team are working to bring it back online. I’ll post more updates here as soon as I get them from the admin team. -Dan J

Update Feb 10th, 12:30m PST: All VPS guests on homie-vserver206 report as being be up and sites should be responding. The restore is running, but we found one particular VPS saturating the NIC was caused the restore to stall a bit. We have moved that VPS in the restore order queue so that it restores last . We will continue to update this post as further updates become available. – Oscar.

Update Feb 10th, 5:30am PST: All VPS guests on homie-vserver206 report as being be up and sites should be responding. The restore is running. We will continue to update this post as further updates become available. – JJ G.

Update Feb 9th, 10:00pm PST: The issues we were having with the backup brick have been resolved. All VPS guests on homie-vserver206 should be up and working normally now. The restore is also running again as well. We will continue to update this post as further updates become available. – Chalise.

Update Feb 9th, 6:23pm PST: The backup machine from which the VPS are being restored is going offline for a short while. Our datacenter team are working on the backup server and will then bring it back online.  Until the backup machine is online the VPS can only load the files that have already been copied to the local drives, so some VPS will likely remain online while others unfortunately will not be offline. We will have more updates here soon. Thanks for your patience. -Dan J

Update Feb 9th, 6:02pm PST: The restore process has hit a hiccup.  The backup machine from which the VPS are being restored went offline earlier. Our datacenter team are working to bring that back online now. It looks like the backup server may be working properly now.  I’ll have some more info in the next few minutes. Thanks for your patience. -Dan J

Update Feb 9th, 5:43pm PST: The restore process has hit a hiccup.  The backup machine from which the VPS are being restored went offline. Our datacenter team are working to bring that back online now. The VPS that were still running directly off the backup server will have probably seen some downtime. We will continue to update this post again as the restore progresses. Thanks for your patience. -Dan J

Update Feb 9th, 2:30pm PST: All VPS guest machines are now up and running off mounted backups via NFS on the new server. The restore is running on all vps guests and some have already finished. If you are missing data it’s very likely that the restore has yet to get to your VPS, but it should soon. We will continue to update this post again as the restore progresses. Thanks for your patience. -Justin

Update Feb 9th, 12:40pm PST: The building of VPS guests is continuing as fast as possible. Currently around 65% of VPS are built and mounting backups. Once this is to 100%, all VPS should be hosting sites normally and the restore from backup can continue in the background. We will continue to update this post again as the restore progresses. Thanks for your patience. -Justin

Update Feb 9th, 9:30am PST: Around 20% of the VPS guests have been created and are mounting backups, and the rest are being created as fast they can. Once this is to 100%, all VPS should be hosting sites normally and the restore from backup can continue in the background. If you are missing data it’s very likely that the restore has yet to get to your VPS, but it should soon. We will continue to update this post again as the restore progresses. Thanks for your patience. -Justin

Update Feb 9rd, 8:00am PST: The replacement hardware has been setup by our admin team. Once the all of the VPS servers for this machine have been built, they will be back up and running off backups. We will update this post again once all the VPS should be running normally from backups. Thanks for your patience. -Justin

This entry was posted on Thursday, February 9th, 2012 at 6:24 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

96 Responses to “Web VPS host homie-vserver206 being moved to new hardware and restoring from backups (RESOLVED)”

We apologize for the omission of the list of affected guests. Here is the list in question:

ps40160 ps40192 ps40196 ps40219 ps40361 ps40468 ps40648 ps40692 ps40717 ps40776 ps40800 ps40866 ps41010 ps41022 ps41074 ps41171 ps41212 ps41383 ps41723 ps41727 ps41821 ps42313 ps42592 ps43068 ps43908 ps43966 ps44121 ps44126 ps44142 ps45946 ps46158 ps46325 ps47317 ps47424 ps47473 ps47626 ps47761 ps47844 ps47907 ps47946 ps48111 ps48181 ps48608 ps48609 ps48610 ps49019 ps49835 ps52512 ps52949 ps53496 ps54356 ps54638 ps55498 ps55512 ps55514 ps56206 ps56795 ps56832 ps65256 ps65563

LOL. There is nothing left to do but laugh at this point. Down again, yay :)

So… which VPS servers does this affect??

sigh…yes looks like down again…Whats the ETA for the stability of the websites? I have customers wanting to get into the website to make purchases and they can’t because of the internal error.

okay, so I’m not on the VPS but I’m experience lag, bouncing and internal errors when my sites are trying to be pulled up. Here are my servers that my account is on:

File Servers:
siegfreid, dabih, teller

Web Servers:
dabih

Email Server:
homiemail-sub5

SQL:
babelfish

The Error the customers are getting is:

“Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@koehlercybercafe.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.”

Really? Again dreamhost, again the servers are down and my sites dont work?? You suck! Do you guys know what you’re doing???

@wildguy: My apologies that this issue has affected you multiple times. Our admins have plans in the works to upgrade hardware and increase stability in the long run. We will be doing our best to get your site back up and running shortly.

@bret: If you wish, you can submit a support ticket and we can verify if you are or are not affected.

@puternutt: This issue is not related to your hosting, you are on shared hosting. I have replied to you via a ticket.

@Bill H: This issue does not affect you, you are on shared hosting. I have replied to you via a ticket.

Our website, has been down since yesterday afternoon during the work day. This is getting ridiculous! Please tell me we are one of the ones affected and that our site will finally return very soon?

@Kris: Your issue is unrelated to homie-vserver206. The issue with the site you linked, is you are using 3rd party dns and it is incorrect. Our support person working on your issue will provide you more specific details shortly.

Thanks, Justin. Will be ready to update the DNS the moment I hear from them.

Once again my sites are all down, with no real explanation.

You guys should be doing this type of upgrading in the middle of the night when there’s the least amount of traffic.

I’ve already sent in my complain to support email, and you guys basically told me “Sorry!” that doesn’t cut it.

Fix your problems people, you’re losing lots of customers.

Are this is related to the fact that our just moved website to VPS is down from yesterday afternoon? Already opened two ticket support about this, no reply. Thanks for any info! (tracking # 4737563)

We need to have the website up asap now. :(

@Kris: You’re welcome. We have multiple people looking into your issue now.

@Jason: As the post said, this was an unplanned outage. We do normally do planned outages or maintenance in the late hours. You are not on homie-vserver206.

@Jean: I’m working on your issue now, reply to come soon.

Is there an ftp outage going on at the moment? cause I can’t access to my ftp in the last days…

@Cris: There is not. If you wish, you can contact support with details and we can assist you.

@Justin: many thanks!
I have a bit more issues, and will report via support.

@Justin, I have not heard anything. Was someone going to email me the correct IP DNS should be pointing to if it has been changed? Or has it been deemed there is more going on than just this? Thanks for your help.

This has now been hours and still no service. This is the second time with the same problem this month.

Not a happy chappie!

@Kris: Looking into your ticket again.

@Kevin: The site you linked is not hosted with us. If you can provide more information, and contact support, we can assist you further.

We moved already. Waste of money here. You guys need some quality and technical training.

@Luis: I’m glad to hear that you have found a host you are happy with. Have a good day.

WoW.. I foud I great day to test how website work on vps..
I wrote to support 3 times.. can that be the reason why I get
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request….

?

@paola: Your issue is not related to this outage. Your custom php was not setup properly, I’ve sent you a support ticket with the details.

please reply ASAP, I need to decide wether to saty w/vps or move back to shared space.. I write from Italy and it’s already 9.24pm, I’m stuck froom 12 hours now.. my collegues are going to kill me (or fire me)

But today, only for a few minutes I saw my site exactly as it’s supposed to be.
I fixed the dns problems,something different happened today

@paola: The response has been sent and the site you mentioned in the URL field in this blog is working now.

Wow, while I was writing, it started working.. did u do something?

Can we please get an update? Coming up on 6 hours with no site. Not good!

Try this site then. The world.edu content is on the VPS server 56206 and that is not working!

@Kevin: Checking into it…

@Brenda: Sorry for the issues still, I updated the post just now. Our oncall admin is keeping an eye on the VPS restore and setup process.

@paola: Yes, I fixed the issue and answered your ticket.

What is the eta of the server coming back up. I have been down for a while now.

mmm…down again..

@Justin The site has been down 24 hours now. We’ve got to get to the bottom of this. We are a medical facility. Please have whomever is in charge of my ticket email me a phone # in which I can communicate to them with. I expected to have this fixed at 9AM. :(

Could not agree more Kris. When the new CEO came on board he made all sorts of smart comments including stupid comments about New Zealanders- great way to annoy your clients. All his newsletters are full of ‘try to be smart’ comments when he should concentrate on getting his service right. I don’t know how many times we have had a problem with this service but is getting to the stage of going somewhere else.

What happened to the Chat service? What happens to the updates here!

Get your act together DreamHost.

@paola: I checked it again, it looks up to me. If you want to submit a ticket with the specific issue you see and how to see it, we can take a look.

@Chad P: Your vserver is restored and your site in the URL field of your comment is loading for me.

@Kevin: Your vserver is not yet restored, but it should be in the queue very shortly. There are only around a dozen still left to create so I am hoping within the next hour or two.

Justin if you think 8 plus hours is acceptable then you should go into politics. This is an awful service.

again???, It is not possible!!!

@Kevin: I understand your frustration, the VPS are building as fast as they can. It has to build a VPS, the OS setup, all services, backup mounts setup, etc. An admin has found a way to increase the speed and they are building faster now, so your services will be restored as soon as possible.

@Justin What’s my update, bud?

@Justin: seems a problem with dns propagating when i add www. in fronf of domain name,i wrote to ssuport, can U check?

@Kris: I had some questions for you in my last response to you. While our devs are working on the issue with so many domains on the single apache, other issues like ftp, what domains aren’t working that are setup, I need more info to address.

okay so Justin, if I am not the only one getting the internal server error and now the 500 error I would suggest that there is an issue with restoring from the NFS and that is pulling resources from others. Strange how the VPS server goes foo bar on you and then shared hosting at the exact same time you start restoring it is when the shared hosting starts to bounce and/or goes down for some. thousands of dollars are being lost during this outage. I know I know you can only do so much so fast, but please don’t take this lightly which I’m sure you aren’t. The sites are down again completely and that has nothing to do with wordpress, caching or of the sorts. that is a connectivity issue plain and simple. Whether it’s a blade, cluster or a plane. Their down. I hope that there isn’t just a couple of people on this. All hands on deck for this, it needs to be resolved and looks to be more than one focus. Could be a coincendence that the VPS dies and starts restore and then shared hosting gets hammered. But not likely that both are being effected like this on multiple servers. Don’t you think it could be lag or connectivity bouncing? I don’t know, I’m speculating from experience, but I’m not there, we’re at your mercy at this point, please keep us posted every second you can. I know you’re busy trying to work on this. Thanks.

All of my sites (shared hosting) are down too with internal 500 errors.

They are NOT on the affected server listed above.

What gives?

@Bill,
Not sure, I just ran a test on my server and it shows the server isn’t there, I opened up a support ticket with the test results in it, so Justin or support staff here is the ticket number for that: #4739924
@Bill, what server are you on?

My test results:
dabih notice Your web server has been rebooted in the last week.
dabih notice Your web server did not respond to our tests. (2-9-2012 @ 3:45pm cst)

@puternutt I’m on manama

@Bill: You provided me no information, but I can say that shared is not affected by this issue. Please contact support if you are having issues.

@puternutt: Your issue is usage related and not downtime related. If you had further questions about the information I sent you, please reply to the ticket.

@Justin, I did reply, please check the information, the server is not responding.

@puternutt: Looking…

My site went down in the last hour as well. Not sure if it’s affected. I put in a ticket: Your tracking # is 4739973. Thanks!

We’re back up, thanks for the help.

@justin, what was the issue, mine are back up too.

@Justin, did you get my last bigger comment? I am not showing it above for some reason after submitting..

Can we have an update please?

@Kevin: Your VPS should now be up and working. I loaded a couple of your sites.

@Phil: I’ll do an update in one moment, getting the last bit of info for an update.

@puternutt & KristinR: Nothing was done by me.

Yes we are back up

@Justin I think comments we are leaving to each other are not being published due to including names or typing domains. Anyway you can shoot me a quick email so I can write back to you and know you’re getting it this time? Thanks.

We may be back up but we are painfully slow

Come on we cannot do anything with the site due to the connection speed. It is your end not this end.

Down again!!!!

@Justin, at 9:15AM, you told me you had multiple people working on my issue. Why is it not fixed? Why have I not received a single email or been contacted by anybody? Being the 2nd day of down time now for our site, this is completely unacceptable. Please have someone, anyone email me immediately to get this resolved! I’m only here another couple hours for today! Thanks.

@Justin I did open a critical ticket 2 hours ago and still no response. The sites are now working but very slow. Ticket # is #4739859

we are down yet again. This is ridiculous!

This has cost us a lot of money so far. What compensation will be made for this farce?

Our site came back up for a brief moment and now has this message: Fatal error: Class ‘vB_Cache_Db’ not found in /home/brendaleann/thebloomforum.com/forum/vb/cache.php on line 128
We also sent in a support ticket (4740147) to get some help with this. We are coming up on 9 hours of downtime….this is a repeat of a few Sundays ago.

when is this going to be resolved?

Update please…..we are still down and have sent 2 support tickets with no response. 9+ hours down

I really appreciate this update. Usually updates get posted offtime and/or after the fact. This time it was very helpful. Thanks!

What update? The 6.23pm comment. Rather than talk about it why not just fix the damn thing. We have now had close to 12 hours with no service. That is simply not acceptable and has cost us considerable revenue and major reduction in current and potential client confidence.

There is no phone number to call and the only number I can find through a whois search goes to an answer phone.

Where is the wise cracking CEO now? Why is he no communicating with his clients?

I sent a support request like 4 hours ago and stated it was urgent. pointed my domain to dreamhost and nothing. godaddy has confirmed that the issue is on your end. why am i still waiting on a response for such a minor issue. is teh server down again or something /// can i please get a response from someone…. ticket #4740160.

I suggest we all look at a class action against Dreamhost for this lack of service, lack of support and it would seem lack of interest in the damage done to our businesse. I have had servers with different providers for 15 years and this is the worst service I have ever had.

It is soooo slow…… why?

I’m with Kevin. I’d be interested in looking into a class action lawsuit for lack of services and support and LOSS OF BUSINESS due to these constant outages. I still am having issues with email (FOR ALMOST A WEEK) and STILL NO RESPONSE TO MY SUPPORT TICKET SUBMITTED AGAIN YESTERDAY!

Does anyone know how to contact Dreamhost? My website has been down all week. The Dreamhost site has also been down all week so I cannot open a trouble ticket. I tried emailing support but the email is rejected if it does not refer to an open trouble ticket.

The service at present is just soo soo slow it is ridiculous. These people will not respond to support requests (I have 4 without response) you cannot contact them, they show complete disregard for their clients.

Their CEO sends out newsletters with smart comments yet he obviously cannot manage a business because Dreamhost is a disgrace.

I am now getting this error message
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@world.edu and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Now down again. This is now over 24 hours of crap

@kevin, I’m getting the same error over and over and I got a response from the admin team stating they found a MINOR issue with the server and fixed it, and they are blaming me for hitting memory cap which they won’t specify also which I am estimating based on my stats that it must be around 20 mb or memory. I don’t see how that is even an issue for a web server of any class. As well as they tell me that I’m pushing limits and that is creating the errors I’m seeing but yet others like you for example are getting the same exact errors on OTHER SERVERS I don’t see how that is a related and I don’t buy it. 100% of my site is cached, I can’t see my media at all and even though my media shows up on FTP it does not show up on the homepage or other pages. I personally think that Dreamhost is going through some growing pains but this was a catastrophic chain of issues with apparently a low staffed week. I’ve been in IT for 20 years going and I know that when the crap hits the fan you are normally left standing alone. Sometimes getting back with tickets, comments and other updates become low priority when troubleshooting and fixing problems. Just keep that in mind. I do agree though, updates need to be posted or commented on, they need someone monitoring the queue so that they can respond and the rest work the issues. I left my last provider because of this very issue 1and1.com always seemed to be down, always email issues and always not willing to support their clients and customers with reimbursement or technical help. I came to dreamhost because of the potential and flexibility. They have a good thing going, they just got to well..KEEP IT GOING…for lack of better words.

The server isn’t ejaculating rainbows at me, I want a class action lawsuit because I haven’t read the terms of service and don’t know shit!

Ha, typical bloody Americans talking about lawsuits. Why not just chill, go outside and do something else while Dreamhost fix whatever is not working.

I don’t understand why the VPS servers are suddenly using more memory. We never used to get these errors in the past. Has something changed?

In the meantime, patience to all whose server is moving. It’s not the end of the world. Class action lawsuits… pah. You are attacking a phantom.

Well we enter an new day (Glan in sometimes pays to check before you make stupid comments as I am not an American). As for Random Emosexual I can having read the urbandictionary definition I can only wish you well in the future and you find your phantom.
I have received apologies from Dreamhost but some 39 hours later our sites are still not operative.

It has taken us 24 months to get the business to a stage where we were gaining traction and 39 hours for Dreamhost to all but destroy it. How do you post vacancy adverts as an example on a site that is not online?

You can forget whatever the terms of service may say because this is gross negligence.

I believe this act of negligence could costs us 7 figures in revenue and lost credibility. Now that is worth a lawsuit.

Heckman (if that is in fact a persons name) why offer me alternatives and then do nothing about it? If you could have moved us to another VPS why didn’t you? Why are we still offline.

Puternutt I recognise what you are saying. I am still amazed that the smart alec CEO has never bothered to communicate with his clients.

Well well they offer you one thing and then…. more BS

Heckman wrote” If you want, you can sign up for a new VPS (which will
NOT be on Homie-VServer 206), then contact us, and we can get your sites
all moved over to that VPS as well as canceling the old one.”

then this morning 12.11 Pacific time
“I sincerely apologize for any issues you’re having with the service.Unfortunately, we cannot move you to another VPS at this time. I sincerely apologize for the inconvenience this has caused you. If you have any other questions, please don’t hesitate to contact us
again.

Thanks! Jen D”

But still nothing done to correct the problem. They are full of it!

Well said Kevin. I’m in a very similar situation although with a fair amount less at stake. I can’t help but feel that dreamhost has all but destroyed my chances of prospering a successful start-up. Throughout this whole mess its the lack of communication that’s really bugged me.

For Gods sake it is now another 4 hours on and still no sites up. What are you doing?

@Random Emosexual: That’s most likely a false positive due to the ongoing restore. Let me know your VPS and I can check on it.

@Adam: What’s your VPS number and I check on it.

@Kevin: I addressed your comments/concerns via email. Please make sure to check your email for my reply. Also, your sites are working at this time.

My sites are still down. Between hacks you have been getting lately, password resets, and downtime, I’m going to start looking at moving all of my domains and sites away from you guys.

So after it took 2 days to respond to my support request all i got was halfass response and a billing reminder. Mind you the the support request was regarding a problem on your end. What adjustments are going to be made to my invoice. adjustment that surely need to be made because of all the downtime this month and last month.

@Rick Dostie: We have only received one support request within the last 8 months and you are not on a web VPS, if you are having repeated issues, please let us know. I replied to you with info about wordpress installs that do not have caching setup.

@Steve Deezy: You are not on this VPS machine, this status post is unrelated to your issues. I apologize for the delay in the last response to your ticket, but reviewing the history it seems resolved as of Feb 10th. If you are still having issues, please let us know via a support ticket. For adjustments for the invoice due to issues, please contact support.

Hi Oscar, VPS number = ps41212. Sites are still down…

@Adam: I have fixed your issues and replied to your ticket. If you have further issues please reply and let me know.

cannot access panel, site responds with Error establishing a database connection…..

this is really frustrating! i’ve been supportive of dreamhost but these downtimes seem like a regular thing now every month.

@joejoe: There is not enough information in your post that would allow me to assist you. There are currently no issues reported with the control panel, nor mysql servers. If you can reach dreamhost.com, you can use the contact form on there to provide details about your errors, and a traceroute, to assist with the panel connection issues.

DH keeps forgetting to list the VPS guest names…..
/*
To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.
*/

Our VPS responds. It pass all of the tests on the control panel. But the WordPress site doesn’t have any of our content! After dealing with the downtime Thursday/Friday, we’ve got a site that is up, but no content. This is definitely NOT an improvement. Our users are getting very upset and there’s nothing I can tell them because this status page says everything is working, just a little slowly.

Hi guys.

Like so many others, my site has been down. Unlike the others, mine has been down going on week 2. This means my email is down, so I can’t write a ticket because I can’t remember my password, and can’t get it reset because all incoming email gets bounced back.

Any help and/or suggestion would be greatly appreciated. My site was down during a lot of interviews for new jobs… which was catastrophic when they were sending itineraries, flight schedules and contracts :/

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Mail bouncing for subset of customers (RESOLVED)
3:06 pm

Mail bouncing for subset of customers (RESOLVED)

Posted (February 7th, 2012 at 3:06 pm PST) by Brian@dreamhost

At this time, we’re seeing some mail bounce for a subset of our customers. We’re in the process of identifying which customers are affected, and what the exact nature of the problem is. As we get info, we’ll update this post. We apologize for the problems this causes! — Brian H

Update Feb. 7th 04:30pm PST: This issue should now be resolved. All mail services are back on and working normally, and any queued messages should now be delivered. If you’re still seeing problems, please contact Support under Support > Contact Support, in your panel. . — Brian H

Update Feb. 7th 04:10pm PST: Ok, we’ve isolated the issue, and are turning mail service back on. For mail that was bounced, the sender will be notified and it will need to be resent. Some mail should have been queued, and in that case, it should be delivered as soon as mail service is restored. We’re terribly sorry about this.

This issue affected mail clusters homiemail-sub4 and homiemail-sub5. You can check if you’re affected by looking at your Account Status in the upper right corner of the DreamHost panel and looking for ‘Your Email Server’. We’ll post more info as we get it. — Brian H

Update Feb. 7th 03:20pm PST: We’ve updated our system to deny relays to promote queue’ing of incoming messages. This means that once service is restored, your mail will be delivered. All previous bounces (sent before 3pm PST) unfortunately will need to be re-sent by the sender. — Brian H

This entry was posted on Tuesday, February 7th, 2012 at 3:06 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

124 Responses to “Mail bouncing for subset of customers (RESOLVED)”

Finally!!! We’ve been having email trouble all day and now cannot send or receive. We’re on homiemail-mx22.g.dreamhost.com

thanks.

Our mail isn’t bouncing but we can’t receive e-mail.

Ditto: we can send, just can’t receive.

Same issues on homiemail-sub5

@Jill, what you’re seeing isn’t actually related, this issue is specific to mail that’s coming into your account, bouncing. Sending is not affected here.

@vinko, same thing!

You guys should submit a support request if you haven’t already, and we’ll get a handle on that for you.

My emails are bouncing and error report at senders end saying user unknown. Emails not showing up in webmail either.

February 7th, 2012 at 3:13 pmMission Control Says:

Dude!

I can’t get or sent emails: please check homiemail-sub4

me too. i can send mail but unable to receive. thx

February 7th, 2012 at 3:15 pmMission Control Says:

My MX records…
mx2.sub4.homie.mail.dreamhost.com
mx1.sub4.homie.mail.dreamhost.com

Apparently there’s an A record for mail as well?

homiemail-sub4 here

Same issue. We can send email but we unable to receive.

That would explain why I’m not seeing any mail. :( I’ve just gotten confirmation mail to me is bouncing. Mail was getting through until at least 12:30 PT. I’m on homiemail-mx23

homiemail-sub4 server, cannot send, can only receive. This is super bush league.

Thanks, Brian! I actually sent an email to support earlier today regarding my “sending” difficulties.

Not sure if it helps, but my company’s having problems too. Able to send but won’t receive.

Email is bouncing to me too.

Same here, my company is also having problems. Able to send but cannot receive.

homiemail-sub4 server here as well. Please fix this soon guys.

I’m receiving the same issue. I have submitted a support ticket for it. My email server is homiemail-sub5.

homiemail-sub5 – can send but not receive

sending emails successfully, but we have not received any emails since approximately 6:30 pm ET (almost an hour ago).
several people have called us to mention that emails they sent to us are bouncing back.

we are in homiemail-sub4.

I am one of those customers! The last email I received was at 2:00pm PST. I am able to send out mail, but I can not receive.

Our incoming emails are bouncing too. Also, almost every day we have a period of time when we cannot send them out. In fact, I usually use the outgoing mail server for a personal account because our dream host outgoing mail is unreliable.

This is about the 10th e-mail issue we’ve had this year. Can’t wait to switch to a reliable service.

Our email accounts our down as well! What is going on over there at Dreamhost? They should send us an email letting us know when they are down… wait they can’t because our emails are down? Nothing better than a client calling to let me know that all of their emails to me are bouncing.

Incoming e-mails from outside domains are bouncing.

Yep, sub4.homie.mail is having this issue too.

Us, too. We can send but not receive. How do I tell what homie whatever I’m on?

homiemail-sub4 still with problems

+1 for homiemail-sub05. thanks.

Us too. Not receiving or sending.

@John Check your DNS records for mail.yourdomain.com

Yep. Just figured out we’re on homiemail-sub4. Can send but not receive. And, it won’t come through on iPhone either.

Same issue here: homiemail-sub4

homiemail-sub3. Can send but not receive.

I am a customers! The last email I received was at 2:30pm. I am able to send out mail, but I can not receive.

February 7th, 2012 at 3:35 pmJesse Miles Miller Says:

homiemail-sub5 also not receiving.

Mail on homiemail-sub4 – have confirmation from clients that mail is bouncing back undeliverable, with messages ‘user unknown’. I was receiving mail up until 2:40pm CST. It seems like I’ve been able to send mail out ok, but do not have confirmation on that.

we have the same problem homiemail-sub4
thanks

Amazing how dependent we’ve all become on email, right? It’s like the world coming to a standstill when you know your business associates can’t hit you with emails.

This is the mail system at host homiemail-mx19.g.dreamhost.com.

I’m sorry to have to inform you that your message could not be delivered to one or more recipients. It’s attached below.

Urgent

Singapore server shagged too. Cant receive email.

Same here. homiemail-sub5. I can send, but can’t receive.

Email senders are getting bounceback from homiemail-mx12.g.dreamhost.com with error messages of unknown user (e.g. mailbox name x123456).

Ticket Lodged, require update please guys! – URGENT!

homiemail-sub5

Still cannot receive emails.

Cant recieve Email, enough with dreamhost, next month moving to different provider…

Can we forward email while this issue is being resolved?

It is working for me now.

We can send, but not receive – we’re on Okul

I received no email since 14:15 Pacific, but just got two 15:33 tests I sent from Gmail.
Thanks for getting things fixed!

Well this is great, just transfer all my hosting to this site and its the 2nd outage in 2 weeks, is this normal for dreamhost?

Dear Dreamhost, can we PLEASE go just a few days without a major outage in which our email and/or company website is down? Pretty please????

Just realized this was affecting my email – some notification emails that should have come in within the past hour plus haven’t. Hope this is dealt with soon, thanks!

Having trouble logging into webmail.

Same problem as above. Can send but not receive.

All my domains are being affected by bouncing/Undeliverable

One test showed webmail could send out to my gmail, but my gmail reply has still not come back in
POP and SMTP are not working at all

I did submit a ticket and hope things will get fixed SOON. After weeks of the earlier homiemail-sub5 / cluster problems, this is getting a bit tiresome…

I’m on homiemail-sub5, but the Undeliverable message going back out says
“This is the mail system at host homiemail-mx22.g.dreamhost.com.”
I’m sorry to have to inform you that your message could not be delivered to one or more recipients…

The incoming email was addressed to yvonne@locdown.com but at bottom of the return error email says:

(expanded from ): unknown user: “x11433576″

Getting the odd email thanks Kristan I checked again after reading your post – but not seeing lots come in and being down for two hours there should be lots of emails being recovered should there not be?

NOW!!!!!!

ERROR:
ERROR: Connection dropped by IMAP server.
Query: SELECT “INBOX”

HAPPEND DREAMHOST??

I’M SOOOO UNGRYYYY

You’ve got to be kidding. I can’t reply to my support request, and my sites are down right now. They’ve been going down and up for the last few weeks, and have been worse these past few days. This is ridiculous.

also on the notorious homie-sub5

Would this affect the mails sent by your system to confirm domain name transfers?

help me. I’m stupid. I continue to stay with dreamhost despite the regular problems that keep happening!

now I have to endure another year because I just renewed with dreamhost.

anyone know of any host that does not present many problems?

Thanks dreamhost
you soured my day

FWIW, I seem to be getting email again. Yay.

@Vinko – I’m with you as part of the stupid club. I moved to Dreamhost from Godaddy thinking I would get much more for the x4 they charge for hosting. I guess you DON’T get what you pay for with Dreamhost.

Why are you people not using google to handle your domain mail? I have been using it here for years, it is simple to set up, and it is rock solid.

I can choose to use a client to retrieve my domain mail handled through gmail or just use the web interface — which is even easier. Spam filters are more robust also.

More importantly, each time DreamHost’s mail delivery goes down, MY email is still coming to and from me without any hesitation.

I bring this up because it is a rock solid alternative to what you are going through right now, and you don’t have to move anything or pay any money for it. Just go in you control panel and set it up by following the simple directions. Within twenty minutes, you can be having worry-free emails.

Emails are now working us again.

DJ. Allyn:

all know that.
but we manage more than 11 email accounts
Gmail charge you X amount when you have more than 10 accounts.

@DJ – Great point! However… some of us own (or work for) larger companies that need more than the 10 free mailboxes that are free with Google Apps.

Considering we have 25 at our firm, we would much rather avoid an extra $250/year for Google Apps, I’d like to be able to rely on the service from my domain provider if that’s part of what I’m paying for.

Migrating off this piece of shit ISP has gone up on my priority list.

This email hosting sucks donkey balls!!!

has then be resolved or not?! i still can not get my emails!

we still can’t receive any emails…… it’s so sucks~~~

While we’re here bitching about not getting our emails, Dreamhost is recruiting new clients via “special events” on their Facebook page.

http://www.facebook.com/DreamHost

I invite you all to share your thoughts on the company’s Facebook page so other people know how often these outages occur. I’ve been standing by and holding back for weeks now, and this has just been the last straw.

How much longer until emails are back up and running again? Cheers

The IT team @ Dreamhost is off this. The little ones are napping now, and the older ones are trying to help Simon Anderson restore his hard drive of pornos that crashed.

Actually, I haven’t received any Dreamhost email since Feb 1. Didn’t notice since I wasn’t getting error messages.
They should just admit openly they want us to leave. It would be preferable to this “slow death” by outage after outage after outage and emails disappearing into cyberspace.

It is not fixed!

ERROR: Connection dropped by IMAP server.

ERROR:
ERROR: Connection dropped by IMAP server.
Query: EXAMINE “INBOX.Trash”

What is going on with this service my mail has been down more in the last 3 months than 5 years with various other hosts.

I cannot access through emil client or Squirrelmail

ERROR:
ERROR: Connection dropped by IMAP server.

Thanks back at it – frustrating when expecting business related emails -

The issue is not resolved. Webmail is timing out and I’m unable to connect to my mail server. This is the second outage in less than a week. I was considering migrating to Dreamhost. That may not happen.

Though it says resolved, still not receiving.

Hello Brain:

we are still can’t receive any emails, and there are still not any error messages show on our screen. The problem are not solved.

Please double check asap!! it’s urgent ~!!!!!!!!!!!!!!!!!!!!!!

This is bullshit. It is not resolved.

Yea the problem seems to be worse ever since this was updated as “resolved” I have not been receiving any email and when going to webmail.mydomain.com I am getting the same as everyone else

ERROR: Connection dropped by IMAP server.

It is 5:10pm, and I can neither send nor receive mail to or from my peacefulglass.com domain. PLEASE FIX THIS SOON.

Email down for two hours now. WTF?!?!?!

Hi guys! The issues you’re having problems with, specifically the “ERROR: Connection dropped by IMAP server.” error aren’t related to issue we had earlier, so submit a support request asap, and we’ll get on those. The issue we had was just for bounced mail. If you’re unable to send, or are getting errors in webmail, that’s a different problem and needs to be addressed by our support staff, so send in a message asap!

It is 9:15pm AST so… 5:15pm PST… I sent a test to myself 10 minutes ago and have not received it. Perhaps there is a delay with trying to catch up with all the email. I have also not received a notification that the mail bounced and needs to be sent again.

According to my web panel, I’m on email server: homiemail-master

I sent a message to my Gmail account and received it just fine.

So… this issue might not yet be resolved?

Keep trying, guys… Valentine’s Day is next Tuesday!

I am still not getting mail sent to my domains.

y mañana? que sorpresa nos esperan?

Hi All, we’re still not up and running here! Help.

Hi,
Its is 5:33pm(1 hour 3 minutes after the notice that everything is working).
Our emails are not working yet. Maybe the problem is not fixed yet?
Brian, can you please update again.
Thanks.

Back up and running

No incoming email since 2:57 pm MST today. We’re coming up on 4 hours of this nonsense. What gives Dreamhost?????

what’s going on??????? why no one response our questions, and solve the problems.
it’s so sucks!!!!!!!!

We haven’t recd any mail since 4:51 cst. This is not resolved! Nothing new in squirrel mail or outlook mail. Help!

May we have an update, please?

I’m sure we oculd eliminate all probems if you guys wouldn’t use such stipd names for the servers. Hardware has feelings, too.

Just got a late night phone call from a client about this. This is NOT fixed guys.

I have mail trickling through. Last mail was from an hour ago. Looks like it’s just taking them a while to get on top of this.

my mail is not working. 2nd time in less than 7 days.

@Hunter: we’d switch to gmail, and it isn’t because we’d have to pay extra for the e-mail addy’s (we’ve got about 25 of them). But my understanding is that google’s current privacy policy states that google can exploit any of the data that goes through their network, and I don’t think that’s good corporate security (for us). I could have misread the policy though, so I’m all ears for any dissenting opinions.

I hear Network Solutions isn’t any better than dreamhost. I have domain names hosted with them. The cost isn’
t much more than dreamhost. I guess I could go with rackspace which is spendy, but I’ve heard problems from them too.

@vinko: I have the same concern about routing all my email through Gmail … I’d prefer privacy AND stability. Does anyone have suggestions for how to achieve that? At this point I don’t mind spending more, I’m a 10-yr DH user but maybe I’ve outgrown them. It certainly doesn’t look good to have to explain to my clients that “my email is down” repeatedly.

What the hell dreamhost ??? Last week I couldn’t FTP, and this weel it’s an email problem. My friend tells me he’s been pinging his sites on your servers and getting 97% uptime. Are you getting too big?

ITS STILL BROKEN!!!! RUN FOR YOUR LIVES!

What is going on here!

I cannot believe that you don’t have more than one authentication server / database for email accounts. Losing emails is not acceptable for any kind of purpose of the hosting.

Dreamhost – where is the update? My emails are not working and your people aren’t updating us???

and tomorrow? what a surprise awaits us?

Fix this please!!! Emails still bouncing.

Please fix this as soon as possible. We didn’t have problems yesterday (February 7) but lost email at 3:00am Wednesday. Have communicated with Tech Support but you guys haven’t added today’s issue to the blog. Reading above I see this is Not fully resolved!

This is not resolved for us. We depend on email. Please fix this now!!

I’d recommend that ANYONE who values their privacy NOT use Google mail. And it’s certainly not just Gmail, it’s Google+, Google Docs, etc.

my god, it’s one thing or another.. time to go elsewhere… bad bad bad service Dreamhost!

This is completly disturbing Ive had my clients on my tail all day for this, submitted a ticket and yet no update, no ticket response, no nothing. Real IT people.

I solicited over 2,000 people last night for genderless sex.

Today they’re all emailing me and they can’t get through!

SEXUAL FRUSTRATION EVERYWHERE!!!

Dreamhost, how can you be so terrible?

You need to buy me a bevy of malefemale prostitutes to take my mind off this error!

Just experienced periodic bounces again! Some coming through, some not.

Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550 5.1.1 : Recipient address rejected: User unknown in virtual alias table (state 14).

Can’t send mail. get error.

using chrome– getting SSL error when I try to log into DH panel. google is warning me to not to proceed. It says:
“You attempted to reach panel.dreamhost.com, but the server presented a certificate that is not yet valid. No information is available to indicate whether that certificate can be trusted. Google Chrome cannot reliably guarantee that you are communicating with panel.dreamhost.com and not an attacker. You should ensure that your clock and time zone are set correctly on your computer. If they are not, you should correct any issues and refresh this page. If they are correct, you should not proceed.”

Firefox says this when trying to go to dreamhost.com —>

” panel.dreamhost.com uses an invalid security certificate.

The certificate is not trusted because the issuer certificate has expired.
The certificate will not be valid until 5/27/2011 8:00 PM.

(Error code: sec_error_expired_issuer_certificate)”

It looks like our sites were down for five hours last night as well.

Does anyone have information on this?

It’s interrupting my attempt to seduce this crash test dummy.

@Evan: That’d be because your computer’s clock is behind by at least six months — as Firefox suggests, it thinks it’s somewhere before May 2011. Fire up the DeLorean, and/or reset your clock. :)

my e-mail accounts still have problems, can’t open a 35 kb mail on outlook via imap, says “can’t download message”

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues with our Irvine Datacenter (RESOLVED)
12:08 pm

Connectivity issues with our Irvine Datacenter (RESOLVED)

Posted (February 4th, 2012 at 12:08 pm PST) by Oscar

We are currently experiencing connectivity issues with our Irvine datacenter. The other datacenters are not being affected.  Our network engineers are currently looking into the issue and we will continue to update this post as soon as we have more information.

Update Feb. 4th 12:24pm: Packet loss from one of our providers was the issue we were seeing. This is resolved now through the joint efforts of our Network team and and our service provider. -Ralph

This entry was posted on Saturday, February 4th, 2012 at 12:08 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

51 Responses to “Connectivity issues with our Irvine Datacenter (RESOLVED)”

I’m getting this error on my site “SQL database connection error: Lost connection to MySQL server at ‘reading initial communication packet’, system error: 113″

Not sure if it’s part of the problem.

Definitely must have every one of my sites there because ALL of them are down.

Please get this gets resolved soon.

Issue is now resolved. I was only able to check @Gnanes domain since it was associated with the account. If you are still having problems please contact support.

Not resolved. Sites still down.

@Eric your site is working for me, what problems are you experiencing?

site ultra slow, no css or images loading, only text. Most other sites OK, one of them has the same symptoms.

If you all are going to use datacenter names in status message then you need to explain to us somewhere how to tell what dataceneter a server is in….. The panel doesn’t tell use in “account status” what datacenter a server is in.

All a status message like this means to a customer is “dreamhost has a problem somewhere so if we are having a problem then that MIGHT be why”

This issue has returned.

Souciant.com is down again. What’s up with this, folks?

All the websites under our server are down. abyssalchronicles.com and neogeoforlife.com, as well as the subdomains associated with them.

Error (PHP-limit) during upload something.
And cannot log into Panel. What`s up?

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@tafelrunde.deutscher-bund.info and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Well done, I hope all can rest today. PS I Knew you would find a solution.

My website, Souciant.com, has now been down for nearly 8 hours. Would somebody at Dreamhost please do something about this? It’s the third day of being down in seven days due to Dreamhost issues. I find the service, and lack of communication, intolerable.

I don’t mean to follow up, but my site is still down – Souciant.com. And, now, when you click through links to articles, via Facebook, in the US, you get porn ads. In Europe, we get redirected to a site called riesparter.com

What on earth is going on at Dreamhost? Souciant has been offline now for eleven hours. This is the first time an outage has resulted in something like this. It would behove you guys to at least contact us.

This issues has not been resolved.

I have been getting emailed for an hour now. Clients are wondering why their sites are down.

We have 19 domains down. I need this fixed now. :)

Thanks guys. I know computers break, especially macs, but come on! :)

Website austinssc.com is down. Internal server error. No support help response. Not sure if it’s related to these issues but it’s been down all day.

Yep, my site is down as well. fourhman.com. “Failed to open page; server not responding.” Went down maybe three hours ago.

neither dreamhost.com nor my hosted sites are available right now… *again*

Connectivity was unstable again tonight @ 7:22pm CST. Multiple clients that were connection to my VPS were Disconnected. Service Interruptions are almost hourly.

I’m getting a “One-Click Error” that says “Can’t connect to the one-click database!” on my WP install – blog.rihanaries.com. Desperate to get this up as it’s been entered into a competition that is currently being judged…

My website appears to be down.

My website is down, I can’t access hosting, and I CANT even redirect my website to another place until this is resolved!!!! Help. Fast!!!!

i get —
Can’t connect to the one-click database!

http://blog.aalzwell.com/

Please resolve ASAP

Icant access the panel at all from Chrome, FF or Safari or IE…

Software error:

Can’t use an undefined value as a HASH reference at /usr/local/lib/site_perl/Ndn/Common/Session.pm line 161.

For help, please send mail to the webmaster (support@dreamhost.com), giving this error message and the time and date of the error.

The control panel is still not accessible, giving the same following error:
Software error:

Can’t use an undefined value as a HASH reference at /usr/local/lib/site_perl/Ndn/Common/Session.pm line 161.

For help, please send mail to the webmaster (support@dreamhost.com), giving this error message and the time and date of the error.

mail sent to support@dreamhost.com is coming back…

Could you please verify?

I have the same error,
I tried to send a mail message but i’ve got a “mailer daemon error”

Error:
Software error:
Can’t use an undefined value as a HASH reference at /usr/local/lib/site_perl/Ndn/Common/Session.pm line 161.

For help, please send mail to the webmaster (support@dreamhost.com), giving this error message and the time and date of the error.

the mail error that i’m refering is trying to inform, the error occurs accessing to panel.dreamhost.com

support mailbox is down.

(expanded from
): Command died with status 255:
“/ndn/bin/support_email_new.pl”

control panel is down.

control panel is down and ftp seems not working …

Are things going to hell again? Cannot access the panel….

The DH panel is down for me. Getting the same error as jrubinart – mail is bouncing sent to you.

Why is it that the only time you guys are down is when I have a second to sit down and work on things? Or maybe you’re down far more often, but I’m so busy I don’t notice?

anyone else have their entire server down at the moment?

Can’t use an undefined value as a HASH reference at /usr/local/lib/site_perl/Ndn/Common/Session.pm line 161.

Still having major issues here in NY office.

This is the end for me at DH, after 8 years…unfortunately I have domains registered there so I have to keep an account until i can transfer them all away.

All my servers are down… what the hell happens again -.-

Can’t access the panel. Sites are up, though.

I cant acces to my sevrer… i contac to u by support@dreamhost.com, but every single mail that i send it to u , i have a delivery mail sistem error…. i dont know what to think. Please solve my problem… i have writing to u from several channels, and i don recive any real response to solve my problem.

Please contac me shortly.

yup my servers r down 2

Site down!

Customers getting: Undelivered Mail Returned to Sender
When trying to email us

My customers are also getting bouncebacks when they try emailing any of my email addresses. This has been happening for half an hour. I’ve sent a support ticket request, and still this status blog doesn’t still reflect that there is a problem?

Same problem as California: My customers are also getting bouncebacks when they try emailing any of my email addresses. Your killing us here!

+1 for incoming emails bouncing back. have submitted ticket. thanks.

Same issue as California – clients are getting bounce-backs on our email and we are not receiving anything – including our internal email. checked squirrel mail and nothing is coming through there either.

Incoming Mail bouncebacks (unknown user from homiemail.dreamhost)… lodged ticket.

Why isn’t this reported as new issue on dreamhoststatus????

Someone told me “connectivity” means “hooking up”…

Well let’s do our part to stop downtime, and hook up for sweating emosex in dark corners!

Who’s with me? I’m not picky… I hope you’re lusciously fat.

@Random Emosexual – WTF! Idiot!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver163 being moved to new hardware and restoring from backups (RESOLVED)
5:07 pm

Web VPS host homie-vserver163 being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 3rd, 2012 at 5:07 pm PST) by Dan J

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver163′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver163’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 6th, 11:30am PST: Thanks for your patience. The move to the new hardware is now finished. All sites should be operational, so if you have any questions or concerns, please contact support with the details. – Andrea

Update Feb 6th, 4:40am PST: The restore continues and has less than 20% left to finish restoring. We will continue updating this post until the restore has completed!

Again, all sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Chalise

Update Feb 5th, 7:13pm PST:Your data is still restoring to the new hardware, and that process is past 2/3rds done. We’ll continue to provide you with restore progress updates, so please check back again soon.

All sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Seohee

Update Feb 5th, 3:05pm PST:Your data is still restoring to the new hardware, and that process is past 2/3rds done. We’ll update again today with the progress.

All sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Andrea

Update Feb 4th, 09:45pm PST: The restore is still moving along. We’re still watching it to make sure things restore properly. We apologize for the delay. — Brian H

Update Feb 4th, 11:10am PST: All VPS guest machines are now up and running off mounted backups via NFS on the new server. The restore is running on all vps guests and some have already finished. If you are missing data it’s very likely that the restore has yet to get to your VPS, but it should soon. We will continue to update this post again as the restore progresses. Thanks for your patience. -Oscar

Update Feb 4th, 4:18am PST: Several of the VPS for this machine are experiencing issues at this time. Our admins are working on getting them back up and running. The rest are running well and off backups, and the restore is running. We will update this post again with the restore progress. Thanks for your patience. -Seohee

Update Feb 3rd, 10:57pm PST: We ran into some issues with the VLAN but got it working and data is once again flowing. The new hardware is now configured, all of the VPS for this machine are up and running off backups, and the restore is running. We will update this post again with the restore progress. Thanks for your patience. -Seohee

Update Feb 3rd, 8:21pm PST: Our admin team are still working hard to get the new hardware configured. Once this is done, all of the VPS servers for this machine will be back up and running off backups. We will update this post again once the hardware is online. Thanks for your patience. -Dan J

Update Feb 3rd, 6:55pm PST: The replacement hardware is still being updated by our admin team. Once this is done, all of the VPS servers for this machine will be back up and running off backups. We will update this post again once the hardware is online. Thanks for your patience. -Dan J

Update Feb 3rd, 5:41pm PST: Our admin team are still setting up the configuration of the new hardware. Once this is done, all of the VPS servers for this machine will be back up and running off backups. We will update this post again once the hardware is online. Thanks for your patience. -Dan J

This entry was posted on Friday, February 3rd, 2012 at 5:07 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

21 Responses to “Web VPS host homie-vserver163 being moved to new hardware and restoring from backups (RESOLVED)”

The following machines are affected:
ps23521 ps23593 ps23671 ps23710 ps23781 ps23831 ps24069 ps24124 ps24166 ps24822 ps25005 ps25140 ps25169 ps25440 ps25663 ps25721 ps27272 ps27629 ps27635 ps27969 ps27987 ps28028 ps28370 ps29012 ps29207 ps34378 ps35860 ps36769 ps36809 ps36812 ps36824 ps36855 ps36944 ps37152 ps39236 ps43161 ps43403 ps43768 ps43942 ps44162 ps46486 ps47085 ps47091 ps58282 ps58886 ps59367 ps72938 ps73862

This is informative. Thank you for the new update. Looking forward to the results.

Also, if any of you are sexy and genderless, please PM me. Since I’m not actually gendered, you don’t have to send pictures of your genitals… just send along a photo of any old thing, as long as it’s attached to you. Cats might count but only in a Platonic sense.

My company’s site is down. I wasn’t notified the site was down after 24 hours. My VPS machine isn’t listed on this outage announcement or the previous. My support tickets are unanswered, so I have no idea if this is even the problem causing my site to be down.

I am so totally disappointed in Dreamhost after being a customer since… 1997? Funny newsletters just aren’t enough to keep me around, folks.

My machine is listed, yet it’s still not working. My site has been down now for nearly 10 hours without ANY notification from Dreamhost. WTF!

Is anyone else missing images or other data uploaded within the last 24 hours?

Update:

We are missing all changes to files on the VPS since Wednesday.

awesome, my websites running again!! thanks guys!

hey so… my dreamhost panel is down.. https://panel.dreamhost.com/ is down.. any ideas when it will be fixed? I cant get to the contact page in the panel to inform dreamhost..

oh panel works now.. nvm..

If it’s working why am I still getting a 404 Not Found error?

I reported some packet loss issues to them this morning for my VPS (25-50% packet loss). Looks like they found the issue and are working on it :)

Over 24 HOURS and still my site is broken. Great work DreamHost, really earning you money this month!

Can’t get to my website.
Disappointed and frustrated.

My website, Souciant.com, has now been down for nearly 8 hours. Would somebody at Dreamhost please do something about this? It’s the third day of being down in seven days due to Dreamhost issues. I find the service, and lack of communication, intolerable.

Our site has now been down for 46 hours. Unbelievable.

I don’t mean to follow up, but my site is still down – Souciant.com. And, now, when you click through links to articles, via Facebook, in the US, you get porn ads. In Europe, we get redirected to a site called riesparter.com

What on earth is going on at Dreamhost? Souciant has been offline now for eleven hours. This is the first time an outage has resulted in something like this. It would behove you guys to at least contact us.

@Joel – We’re checking into your http://souciant.com/. Please note, it’s not related to this outage, you’re not on the affected machine. Your site has been hacked, you will receive an email from us soon with detailed account scan results.

@Trey, Ange @ GnomeAngel, – your domain is loading now. Sorry about the delay.

@Marc Gunther – you’re not on the affected machine and your site is loading. If you need further assistance, please contact support through the DreamHost panel.

@Thom – looks like you were already sorted, sorry about the delay.

Please keep an eye out for updates, but if you’re having issues with your sites, contact support through the DreamHost panel, so we can assist you. All sites should be operational.

Please can someone update me on the status of my site. All I get is a blank screen for the website and the dashboard for http://www.pirc-ny.com.

This has been down nearly all day. I checked about 3pm EST and it was down and has been down the whole day. I can’t tell if I’m on an affected machine but I would think after over 8hrs it would have been up.

I put a ticket in to support but have not heard as of yet. Can someone please let me know if I’m on the affected machines so I at least know once they come up this will be resolved.

Not sure what has happened over the last 2 months since this is so unusual for you guys. Please keep us apprised.
Thanks
Barbara

The last two weeks all my websites have been down several times a day. The chat has been down every second of the day during this time, and on support they have me sending them tracing routes etc as if it is my location which is the problem.

Im disgusted, really disgusted, unless they sort this out now I will have to move my sites somewhere else – something I rather avoid as there are 150 of them. Also, it is extremely annoying that they dont allow you to create separate accounts for bulks or even each domain if you want. Now, when one page is down ALL are down. Clients quickly check out other websites within my network and conclude that it is my company which is the problem, not the server.

@Barbara – your support request is being forwarded to the team who can help you with your hacked issue (that’s why your site is not loading, you’re not affected by this posted case)

@Thomas – we apologize for the issues you have been experiencing. I’m unable to check on your case without your details. As the status post states, there should be no downtime in this case for days now, so you may have some other problems that need to be looked into. Please contact support through the DreamHost panel, so we can assist you. You can also request a manager review of your case regarding separate accounts. We’re here to help and would like to get a chance to give you a hand with your case.

♥♥♥

AWESOME!

Thank you to Dreamhost Staff who restored our files!!!

♥♥♥

♥♥♥

But the panel’s down now?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver168 being moved to new hardware and restoring from backups (RESOLVED)A
8:54 am

Web VPS host homie-vserver168 being moved to new hardware and restoring from backups (RESOLVED)A

Posted (February 3rd, 2012 at 8:54 am PST) by Oscar

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver168′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver168’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 6th, 11:30am PST: Thanks for your patience. The move to the new hardware is now finished. All sites should be operational, so if you have any questions or concerns, please contact support with the details. – Andrea

Update Feb 6th, 4:55am PST: Data is still restoring to the new hardware and there we are now waiting on the final 15% to complete.

As previously noted, all sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Chalise

Update Feb 5th, 7:11pm PST:Your data is still restoring to the new hardware, and that process is past 4/5ths done. We’ll continue to provide you with restore updates, so please check back soon.

All sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Seohee

Update Feb 5th, 3:00pm PST:Your data is still restoring to the new hardware, and that process is past 3/4th done. We’ll update again today with the progress.

All sites should be functional, as we’re running them from backups. The same files that you see now on your site will be the ones showing once the restore is complete. You can upload files now, and they will not get overwritten by the old version. If you have any questions or concerns, please contact support with the details. – Andrea

Update Feb 4th, 9:50pm PST:The restore is still moving along. We’re still watching it to make sure things restore properly. We apologize for the delay. —-Brian H

Update Feb 4th, 11:10am PST: All VPS guest machines are now up and running off mounted backups via NFS on the new server. The restore is running on all vps guests and some have already finished. If you are missing data it’s very likely that the restore has yet to get to your VPS, but it should soon. We will continue to update this post again as the restore progresses. Thanks for your patience. -Oscar

Update Feb 4th, 4:03am PST: All of the VPS are up and running from backups and the restore is at 19%. We will continue to keep you updated on the restore progress. -Seohee

Update Feb 3rd, 11:23pm PST: All of the VPS are up and running from backups and the restore is running well. We will continue to keep you updated on the restore progress. -Seohee

Update Feb 3rd, 9:45pm PST: The  replacement hardware is online now. All of the private servers appear to be running now mounting files from the backups. Thanks for your continued patience. The files are currently being copied to the local drives. So far about 3% has been copied. We will continue to keep you update as information is made available. -Dan J

Update Feb 3rd, 6:17pm PST: The  replacement hardware is online now. All of the private servers appear to be running now mounting files from the backups. Thanks for your continued patience. The files are currently being copied to the local drives. So far a little over 1% has been copied. We will continue to keep you update as information is made available. -Dan J

Update Feb 3rd, 4:17pm PST: The admins have the replacement hardware online now. So far they have been able to bring about 9/10th of the private servers online mounting files from the backups. They are working to get the remaining VPS online as well. Thanks for your continued patience. We will continue to keep you update as information is made available. Please check back! -Dan J

Update Feb 3rd, 3:01pm PST: The admins have the replacement hardware online now. So far they have been able to bring about 1/4th of the private servers online mounting files from the backups. They are working to get the remaining VPS online as well. Thanks for your continued patience. We will continue to keep you update as information is made available. Please check back! -Dan J

Update Feb 3rd, 2:01pm PST: Our admins have ran into a few problems while attempting deploy the new hardware. They are really working hard towards restoring the service. Thanks for your continued patience. We will continue to keep you update as information is made available. Please check back! -Brian S

Update Feb 3rd, 1:45pm PST: Our admins are diligently working towards restoring service. We ran into a slight issue while trying to mount the backup machine to the new hardware. This has been corrected. The NFS mount has been successfully created. The machine will be online shortly.

If you are wondering, what exactly is a NFS mount? Here is your answer. The NFS mount allows users to access files across a network and treat them as if they resided in a local file directory. In this case, the new hardware is accessing your files that are stored on the data archival machine via the NFS mount.

Hopefully you have a understanding of the process. We will continue to keep you updated. Please check back for up to date information! -Brian S

Update Feb 3rd, 12:04m PST: Our admins are diligently working towards restoring service. Everything will be resolved shortly. The admins estimate that the machine will be back online within the next 20 minutes. We will continue to update as information is made available. Please check back for updates. -Brian S

Update Feb 3rd, 10:45am PST: The new hardware is up now and our system administration team is almost done mounting the backups via NFS. Once this is done, all of the VPS servers for this machine will be back up and running off backups. We will continue to update this post as this process continues. – Oscar

This entry was posted on Friday, February 3rd, 2012 at 8:54 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

10 Responses to “Web VPS host homie-vserver168 being moved to new hardware and restoring from backups (RESOLVED)A”

The following machines are affected:
ps23373 ps23397 ps23496 ps23526 ps23563 ps23640 ps23644 ps23677 ps23715 ps23787 ps23918 ps23960 ps24007 ps24303 ps24339 ps24419 ps24484 ps24697 ps24703 ps24711 ps24719 ps24752 ps24889 ps24911 ps25479 ps25644 ps28450 ps29119 ps29277 ps29289 ps29375 ps29736 ps34452 ps34465 ps34645 ps37251 ps38207 ps38339 ps38525 ps39096 ps39233 ps39427 ps40054 ps46584 ps46662

Is there an update here? It’s been an hour. We’re still completely down.

1:40 PM PST and we’re still down. Almost done?

Our site has been down since 15.30 GMT, over six hours. When will this be fixed, to be without service for so long is unacceptable and very damaging to our business

Ring ring! Hello, Dreamhost? We understand that getting hacked has made you turn off your live chat . . . but not being very open about WTF is going on isn’t helping.

Any new estimated time? I am losing revenue and my sites been down all day.

@Marsh:

Your linked site is up. Actually looks like fun, this MMORPG thing.

@Everyone else:

Since we’re all here not doing anything with our web sites, let’s get naked…

Feels like the lottery. I wonder when my number will be up.

Please keep an eye out for updates, but if you’re having issues with your sites, contact support through the DreamHost panel, so we can assist you. All sites should be operational.

♥♥♥

GLAD TO BE BACK UP

♥♥♥

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘portvila’ not accessible (RESOLVED)
10:38 am

Shared web server ‘portvila’ not accessible (RESOLVED)

Posted (February 2nd, 2012 at 10:38 am PST) by Oscar

The shared web server “portvila” is currently not accessible. It did not reboot correctly after having its raid card replaced. Our system administration team is working on the issue and we will update this post as soon as we have further information.

Please note, this is regarding the shared web server ‘portvila’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. – Oscar

Update Feb 2nd, 10:40am PST: The raid array has finished rebuilding and all sites and services on ‘portvila’ are back up and running normally. Thank you for your patience! If you have any further issues, please be sure to contact support Support > Contact Support, in your panel. – Oscar

This entry was posted on Thursday, February 2nd, 2012 at 10:38 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Shared web server ‘portvila’ not accessible (RESOLVED)”

Our site is down and so we can’t connect to FTP either. http://www.onegreenplanet.org
Please reply !

It’s still down at noon PST (2:45 ET)

It’s still down here also

We fixed a lingering issue with this and *now* you should all be back up. Please submit a support ticket if otherwise!

The site is up and running, but the index page has dropped an image. I would like to reload the index page, but I still can’t get into the server with the proper ftp login info.

So glad I switched hosts.

Let me write the perfect DreamhostStatus.com message:

(1) I’ve been down for four hours, and lost over $500!

(2) My clients are calling!

(3) I’m gonna leave you for spry.com, Host Gator, Bluehost or Rackspace! OR ALL OF THEM.

(4) I pay $8/month for this, how can you betray me?

(5) You’re incompetent, probably drunk, can’t Linux, etc.

(6) I’m calling my lawyer and I’m going to sue for breach of bandwidth and unconscionable delay!

(7) I could have hosted this site better on a Windows 3.1 machine with a web server written in BASIC A!

(8) Dreamhost used to be a dream, but now it’s the work of Satan. Log off now, it’s too late for me!

Did I get them all?

It’s out again. It’s been going out all day. This is highly inconvenient.

It’s still out. Fix it. This is terrible service.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Email connectivity issue in one of our datacenters (RESOLVED)
8:37 am

Email connectivity issue in one of our datacenters (RESOLVED)

Posted (February 2nd, 2012 at 8:37 am PST) by Oscar

We have received reports of connectivity issues to our mail service in one of our datacenters. It manifested in slowness when sending or receiving and mail timeouts from both webmail and email clients. Our system adminstration team is investigating the cause and is working diligently towards a resolution.  We are already seeing improvements, but we will continue to monitor the situation as long as necessary. We’ll update this post when we receive further information. Your patience and understanding is appreciated, and we apologize for the inconvenience.

Please note, this did not affect mail sending from web servers, or any other service such as web and mysql. – Oscar

Update Feb 2nd, 9:13am PST: The cause of the above problems appears to be a slight connectivity ‘hiccup’ between the file storage machine and the email processing servers. We double checked for hardware issues, and it checks out fine. We will continue to monitor the connection and investigate more about that failure.
We’re resolving this case, but if you need further assistance, please don’t hesitate to contact support with the details, so we can assist you. If you already emailed about this, you will receive a reply shortly. – Andrea

This entry was posted on Thursday, February 2nd, 2012 at 8:37 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

12 Responses to “Email connectivity issue in one of our datacenters (RESOLVED)”

After 7:15AM I have been unable to send email from my local client. I can send from webmail, and I can receive mail.

@Ken – are you still unable to send from your email client? We have tested some cases and are not able to replicate the issue any more. We would be happy to look into your case, so please submit the details (the email address or addresses affected and your SMTP settings) in a support ticket through the DreamHost panel.

While it was down, I was not able to get into webmail.
However, all is better now. Thank you.
Jim

I’m unable to connect to FTP now — 12:44 PM EAST 2/2/2012

Is this related?

Same as Nathan. Cannot connect to FTP. It is saying “login incorrect”

Still slow… But working

Hi. Mail has been restored to my domain, but there’s a bunch of missing e-mail that is received through that domain. Will I see that mail catch up at some point? I’d submit a support ticket, but I’m at work without my web panel login info.

Thanks.

-Dan

Kudos to the people in this thread. You’re all polite and rational. Am I dreaming?

um, i logged into webmail, clicked msgs, got logged out with some error, went back in, and the messages are gone. WTF!!!!!!!!!!

Seems to be down again, I have been checking about once each hr, I did catch it working a bit but is down again. All FTP transfers are really really slow as well, hoping you can get the systems working one day.

I’m unable to send mail as well.

No webmail access, no access through my mail client. Been happening over the last 4-5 days. If I try to access webmail: “Page not found.”

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Scheduled emergency network maintenance tonight at 8pm PST (RESOLVED)
7:45 pm

Scheduled emergency network maintenance tonight at 8pm PST (RESOLVED)

Posted (January 30th, 2012 at 7:45 pm PST) by Seohee

Network maintenance is scheduled to start at 8pm PST tonight to update some routes as they are causing additional problems during our network outages. This should cause no downtime at all, but please do contact support if you run into any issues.

Update Jan 30th, 10:04pm PST: The maintenance has completed now. The admins are currently verifying connectivity and route advertisement. We haven’t seen any reports of downtime but if you are having trouble then please contact support. -Dan J

Update Jan 30th, 9:00pm PST: The maintenance started about an hour ago. Things are proceeding slow and careful. Our network admins are confident that this shouldn’t cause any downtime, but if you do have any problems please contact support. -Dan J

This entry was posted on Monday, January 30th, 2012 at 7:45 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses to “Scheduled emergency network maintenance tonight at 8pm PST (RESOLVED)”

So….downtime is being experienced right now. And “contact support” doesn’t work b/c it hangs after the first screen. Are there any updates on outages? Mine started about 1 hours ago not being able to connect to the database.

My SQL server will not connect. I tried to submit a ticket but the support page of the panel won’t load. Help!

All functions of the panel are broken due to the fact that the HTML code that creates the “CRITICAL ANNOUNCEMENT! PLEASE READ!” messages on all panel pages is screwed up.

The support screen is broken and I desperately need to fix my git repos since the network hiccups cause corruption.

Panel is broken by the “Critical Announcement! Please Read!” message.

I went down to the ghetto today. I got gang-raped by violent thugs for FOUR HOURS.

I’m going back tonight.

January 30th, 2012 at 8:15 pmJason C. Levine Says:

If anyone from DH is watching this, there is an unclosed tag in the panel header so no one can submit tickets (or much of anything else).

Sorry about that guys! The panel should be working now, please open a support ticket if you see any more problems.

Why is HTML not stripped from comments?

Thanks for the update.

This is getting unreal!

I need the website to go up! I can’t keep having my website down! Jizzz!

January 31st, 2012 at 2:21 pmhello? anybody home? Says:

mail from web forms is down. why is there no status update about this?

no status updates about current status down.

Our company is experiencing rolling blackouts with our e-mail. Have been for weeks. How can I ever recommend Dreamhost with your current level of service?

We just… migrated from here. This was too much. Will never be with dreamhost again.

@Luis:

I can’t reach your website. I get a 503.

have been down 2 days!!!!!! no phone to call????????

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server Sinistra having issues (RESOLVED)
1:16 pm

Shared web server Sinistra having issues (RESOLVED)

Posted (January 30th, 2012 at 1:16 pm PST) by danwy

The shared web server ‘sinistra’ is having some issues at the moment. It seems like there may be some php packages missing on it due to the recent issues we’ve been having, but we’re not certain yet. This is causing sites hosted on it to give 500 errors and give ‘premature end of script headers’ errors in the domains error logs. Our admins are checking on this server and are working on getting it back up and running smoothly. We’ll update this status post shortly once we have some more information regarding its stability restoration process.

Please note, this post is only regarding the sinistra web server. No other web servers or other services, such as email or mysql have been affected. To find out if you’re on this server, please to to the panel and check the machine listed for your FTP/Shell user. – DanW

Update Feb 1st, 4:02pm PST: We’re happy to report that sinistra has been stable for a while now, and its load is in the normal range. We’re confident that the offloads have helped to resolve this problem, but we will continue to montior the server for a little while longer.
We apologize for all inconvenience the instability has caused. – Andrea

Update Feb 1st, 11:34am PST: Our system administrators continue moving users off this server to alleviate the load that has been slowing things down lately on your websites. They are also closely monitoring to find balance between how fast they can offload to lower the load and not cause too much strain at the same time. Unfortunately, the machine had to be rebooted again earlier this morning, but it has been stable ever since.
We appreciate your continued patience. Please check back for updates. -Andrea

Update Feb 1st, 5:57am PST: Our Admins are still working to move a few users off this server and have been monitoring it to keep the load down during this process. We should have more updates to this issue later this morning! Thanks for your patience! -Patrick

Update Jan 31st, 8:12pm PST: Our admin team are still battling with the load a bit. For the most part it seems stable now, though it spikes up occasionally. We are still in the process of moving users off of the machine. A bit of a load is expected as it’s caused by the moves, so please hang in there! We’ll update this post with further information as it becomes available. -Dan J

Update Jan 31st, 4:30pm PST: We are still in the process of moving users off of the machine and see that the load and performance are greatly improved at this time. A bit of a load is expected as it’s caused by the moves, so please hang in there! We’ll update this post with further information as it becomes available. -Seohee

Update Jan 31st, 1:30pm PST: We are continuing to work on getting this machine stable, and are currently moving users off of the machine. Things should be improved at this time, and we’re still working on getting them back to normal. We’ll update this post with further information as it becomes available. -Brandon

Update Jan 31st, 11:52am PST: We are continuing to work on this machine to keep the load at an average. We firewalled some offending IP addresses and are moving off some customers as well. The latter may cause a tiny bit of strain due to the copy processes, but we’re looking better. Thank you for your patience. – Andrea

Update Jan 31st, 9:00am PST: We are close to a solution. We’ve decided that the best course of action would be to move heavy resource users off of the machine, which is being performed shortly. We believe this should help tremendously. Thanks again for hanging in there wish us. Your patience and understanding is greatly appreciated. Please check back for updates. -Brian S

Update Jan 31st, 9:00am PST: Our admins are still investigating the load spike. We have disabled a few heavy resource users. However, the stability issues continue to resurface. The machine was rebooted a few minutes ago. We will continue actively monitor until we identify the source of the stability issues. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated. -Brian S

Update Jan 30th, 5:24pm PST: Our admins are still investigating the load spike. They have isolated the cause to a few users that are over consuming system resources. We hope to regain stability shortly. Sorry again for the problems that you have experienced. Your continued patience is greatly appreciated. Please check back for updates. -Brian S

Update Jan 30th, 4:12pm PST: Our admins are still investigating the load spike. They have isolated the cause to a few users that are over consuming system resources. We hope to regain stability shortly. Sorry again for the problems that you have experienced. Your continued patience is greatly appreciated. Please check back for updates. -Brian S

Update Jan 30th, 2:15pm PST: Our admins are still looking into the stability issues. There are a few overloaded apache instances on the machine, which is currently being investigated. The admins have also pinpointed a few users that are consuming more than their fair share of system resources. This is being addressed as well.

We will continue to post updates as information is made available. Please check back. -Brian S

This entry was posted on Monday, January 30th, 2012 at 1:16 pm and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

75 Responses to “Shared web server Sinistra having issues (RESOLVED)”

Hey, I just used you guys on a recommendation to purchase a domain and just noticed you guys have sooooo many issues. Now I cant even log in to view any purchase info and my card has been charged.. and the website is not available from other sources… what do I do now?? lol

How about you fix the other outage issue on the VPS servers first? As it’s been almost 2 days now!

I just lost a client, please pro-rate my service

Can you tell me when I will be up and running again? Thanks.

This is the worst service ever.
Third day in a row.
Contacting the supports yields an email on how sorry they are and that an administrator restarted “my” (as if they are my machines) machines.

If anyone is reading this prior to purchasing a service from DreamHost. Please reconsider !!!

@Doron

Unfortunately… too late for me :(

“Sinistra server having issues”

Oh, really??????? buddy you should be awarded a nobel prize for that statement ¬¬

@Alex:

Unfortunately… too late for me

I can help you work on those premature ejaculation issues, bro. Or is it… sister? No worries, I’m wired for both!

5:47CST and all is well. Thank you for the effort in restoring such a huge mess.

@Alex – your domain is registered and your account is approved. You have no hosting service though, so you’re not affected by this outage. Could you contact support with the details of your login issues? If you just can’t access the DreamHost panel, there is a forget password link at login.
@Sad Client – are you on a VPS that still needs attention? We left that status post active for reference, we are picking up lingering cases that have customizations and couldn’t be fixed automatically. Please contact support through the DreamHost panel with details, so we can assist you.
@Paige – I replied to your support message
@Fran – the site you linked to your post is up and running, was there another one you needed help with? Please contact support with the details.
@Doron – Unfortunately, I can’t tell what you’re referring to, so I’m unable to assist. We’re answering tickets individually, please contact support again if you’re still having issues.
@Jim – sorry about the inconvenience, and thanks for your kind words.

I just moved to aws in the first week of January. Rock-solid since then.

smooth, all support via panel no longer active….I am on server apparently “not affected” by all the problems lately

The contact support page is blank. I’m trying to notify them to look into our site which is down, generating an Internal Server and 503 error. Our Account Status looks fine, though.

I sure do love sodomy!

I have an email I set up this evening it can send but not receive, if I log into email manager i can enter username and password, but if i click secure or insecure connection it throws an error…username or password incorrect….come on this is really stressful and can’t open a ticket…my server isn’t listed as being affected

LOL No wonder they disallow all search engines from indexing ANYTHING in this entire domain!

Look at the DAILY crashes!

Please, people, tweet about the sh*tty service you get here, comment on forums, blogs, everywhere and point to this domain as proof.

I’m amazed that I constantly see people saying they lost clients from DH outages.

You people are seriously using $10/month web hosting as the backend for your paid web design contracts!?

Sounds like your clients dodged a bullet.

I’m still getting error 500 in my sinistra server.

Is this issue still open?

@Francisco, of course it’s open. All my sites are down, again (no surprise though ¬¬)
@ian, you should know that when you’re developing web apps and webpages, some clients ask for an overall low cost product, whilst others are interested in stability & availability. Not all the customers are the same.Their needs vary from one to another. So if a customer is on a low budget, you have to offer him choices. Maybe he/she can’t just afford the costs of a solid rock hosting. So they’ll have to accept some inconveniences if they are only looking for costs, but what had happened (and happening yet) this last days is beyond justification. I’m utterly dissapointed with DH. I do respect all opinions, but this is mine. I’m gonna migrate all of my “low cost” sites out of DH asap.

I can’t get anybody to respond to my requests for support. what can I do? Am I just screwed?

My sites it’s not work, html + flash

Sites are still slow or not connecting … ftp is still slow … going to be out of business at this rate plz get this resolved!!!

after 20 hours the server still not working! I’m moving to another provider

MY SITES ARE NOT RESOLVING. NO ANSWER AT ALL. THIS ISSUE PERSISTS SINCE LAST WEEK

Message from you (Jan 31, 2012 – 07:36:58 / #4718480)
Subject: DATABASE IS DOWN URGENT
We are in a presentation for a very important client,
and the databases are DOWN, we open a ticket about this 9 hours ago with NO REPONSE !!!

we need a solution for this NOW.

Message from you (Jan 30, 2012 – 22:08:30 / #4717767)
Subject: mySQL is down
Hi there guys, well, how can I say it…

Where´s our mySQL Server? Yesterday we had problems with our VPS (apache didn´t work, ftp etc) and now the mySQL Server isn´t working. Can you see this?

I have SLA contracts that expect you to follow your’s
any solution on going ?

any news on the tickets ?

I have multiple domains on ‘sinistra’. I can get to some, and some I cannot (via http). I’m not a huge customer, but I am a networking professional and the fact you do not have high availability configured is astonishing. If this isn’t resolved today (hours), you just lost all my domains, future business, and referrals (not to mention, you’ve created another anti-advocate). I would’ve posted a ticket, but that page is mysteriously nowhere to be found. Seems amateur. This may be motivation to migrate to another host, either way. Update needed now.

January 31st, 2012 at 9:30 amStupid to be with Dreamhost Says:

ALL OF MY VPS SITES STILL DOWN, maybe you forgot it has been so long.

How was/is your vacation going?

When will an VPS admin be back at work? Quite, Please don’t wake the shared hosting people.

Just a proposal…. why don’t you change the name “Sinistra” to “Fukushima”????

I am trying to work on the site http://www.cosmowave.in and it drops continually. I have a deadline and my work can only be done through the internet. Please, help!
Nikos Vlantis

Dudes, this is worst than I could imagine of. Now january seems to have more than 300 days… the outage affects the whole solar system. —->”Update Jan 301st, 9:00am PST”

@Hamilton, I will take a look at your support ticket.

@BA, Thanks for the suggestion. However, I do not believe it would help in this matter. Since a particular service is not failing. Nor is a failover required.

@Stupid to be with DreamHost, This status post is regarding a shared server. Please contact support with the details regarding your VPS

@Ruben, We share your frustration. Remember we are in this together. Our goal is to provide a great hosting experience for everyone. We are diligently working towards regaining stability.

@Nikos and All, Until we have identified the source of the problem, there will be intermittent outages. There are 2 admins actively monitoring the machine at the moment. Please continue to bear with us. Your continued patience is greatly appreciated.

@brians

Thanks for the feedback.

We are all frustrated and some are even very angry, but any feedback is always worth.
Even if it is “we are trying hard”.

My website is still failing, but with a smaller frequency than a couple of hours ago.

@Brians
Thank you for your response.
I don’t know if it helps to notify you that I am using a translation script, trying to improve the quality of the automatic translation of the wordpress site cosmowave.in hosted by Christos Drougas, and it is during this procedure that the server is failing and then the website is down and I get an “Internal Server Error” whenever I retry to open it.
My problem is that there is no other way to do this job (I have to be online) plus, that I ought to deliver the job tomorrow (I am in France, so count +8 hours from yours), which means that I just don’t have any more time left.
I would appreciate an estimation: to know where I stand, so that I can reschedule
Nikos

another one #4718926

sure wish we could FTP into our site.. have a critical issue to address and have been unable to access, not to mention our entire day of being down a few days ago. really making it easy for us to switch hosts.

My site is still dead with 500 errors for the last 2 days and no ticket response. Do you guys really suck this bad? Did your password loss include root access to your servers or something too? 2 days of downtime is the most rediculous thing I’ve ever heard of esp with no offer of any sort of compensation. Just a we’re sorry but we really suck at our jobs as much as you are mad at us. We share your frustration……thanks for nothing.

January 31st, 2012 at 11:29 amwow you guys are filled with fuck Says:

PANEL.DREAMHOST.COM IS DOWN !!!

Oh for fuck sake, HIRE SOME FUCKING COMPETENT I.T. GUYS.

January 31st, 2012 at 11:36 amwow you guys are filled with fuck Says:

wget panel.dreamhost.com
–2012-01-31 11:35:16– http://panel.dreamhost.com/
Resolving panel.dreamhost.com… 208.97.187.201
Connecting to panel.dreamhost.com|208.97.187.201|:80… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://panel.dreamhost.com/ [following]
–2012-01-31 11:35:16– https://panel.dreamhost.com/
Connecting to panel.dreamhost.com|208.97.187.201|:443… connected.
HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.
Retrying.

–2012-01-31 11:35:17– (try: 2) https://panel.dreamhost.com/
Connecting to panel.dreamhost.com|208.97.187.201|:443… connected.
HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.
Retrying.

–2012-01-31 11:35:19– (try: 3) https://panel.dreamhost.com/
Connecting to panel.dreamhost.com|208.97.187.201|:443… connected.
HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.
Retrying.

–2012-01-31 11:35:22– (try: 4) https://panel.dreamhost.com/
Connecting to panel.dreamhost.com|208.97.187.201|:443… connected.
HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.
Retrying.

Panel is down for second day in a row. It works, then doesn’t. Works, then doesn’t. What is going on? I’ve got clients asking me questions that I can’t answer. Been with you guys for almost 8 years. I’d like to make it to 9.

I can acces the fTP but i only see the “logs” and “maildir” directory. Plase, how can i see the rest of my site to update it? Thanks

mysql is down.

Load average on sinistra leapt to 1500+ and websites aren’t responding again :-(
Please find a solution, I like DH and I want to give it another chance.

January 31st, 2012 at 2:21 pmhello? anybody home? Says:

mail from web forms is down. why is there no status update about this?

my website uploaded half and images are half displayed and the menus won´t even show. please get this done asap.

ps60484

has a VERY slow response time.

Does anyone else use Moodle on their site? If so, is it working for you. I can access all of the webpages on the site but cannot access Moodle. It has not worked since the outage.

My account page is messed up on the panel. I had a bunch of domains, goodies, etc. All of these are gone. The services seems to be working. I cant change my shell password and my account status says “pending”. I am unable to find the customer email address or call back number. Clicking on callback doesn’t seem to work

i switched over to E2 Hosting. Use pixelpost for 30% off. You might want to check them out! DH, you suck!
https://bitly.com/cheaphosts

I just lost a client…

He fell into my rectum.

My site was working yesterday but it’s down again… do you guys have some sort of ETA for when this problem is going to be fixed?

Thanks!

I just hate how slack you guys are about the support tickets.
I have a website on and i will hjave it for only 3 months.
It has been 1 week of problems since I had it on and it still with problems!
Whe will it be solved?! It is amazing, I already contract the VPS service to become more independent and I’m still experiencing the same problems!
Amazing! I will give to the end of this week, If i still have problems I want my money back and will migrate to another server.

Can we talk @#$@# about local hosting companies here?

Like Houston hosting companies that cover up their internal hacks and blame it on misconfigured disk drives, esp. when they got owned by Windows trojans?

Hello:

my site is up and running as per today, everything works fine. hopefully it keeps like this.

goddamn…… down again!!! (why am I surprised? I shouldn’t be)

All my sites are down. Not on this server though, on Kramer.

Yeah… same for me.. all my sites on kramer are down. what a fuckup…

all my sites are down, including our flagship client!

ps27825 is the web server

Please help!

We’re back up now. Thanks dreamhost!

ALL SITES ARE STILL DOWN AND I HAVE RECEIVED NO RESPONSE. ps27825 is the web server

Does anyone know if ps40761 is also down? I can’t seem to get it to work!

Well that is it, I kept saying I would do it but kept giving them chances. My site was down again today, I just can’t take it anymore. I’ve signed up for another VPS service and will be moving my files over. And I was paying for VPS service.

site not live on wasat, but dns points correctly. anyone else?

Webmail is slow as molasses.

websites are up again, but all my emails are down now. webmail too!…. and I dont’ remember when I last saw the Live Help actually being Live.

I’ve signed up for another VPS service and will be moving my files over. And I was paying for VPS service.

That’s nothing. I cut off my genitals and moved to HostGator.

My emails aren’t working – I’m on ruchba – what is going on? I am missing all emails going back to when I started the account with Dreamhost last year october.

Something’s definately wonky with some the email servers. I’ve lodged a service ticket but haven’t heard anything back as yet:(

Emails back on now:)

Resolved?????

Not Working my Sites.

Error Message.

“Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.”

I have realized the siniestra server is very slow and hard to upload by nights, and also in the very morning is very fast, it is just that I don´t want to upload at 6:00am everyday.

my site will not load, its obviously siniestra server related, there is only so much I can tolerate of this, this has been an awful experience.

@Lorcan:

It’s loading now. There might be a problem though, since your whole site is in English. Do they speak English in Ireland? I thought you spoke Inuit.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues in one of our datacenters ( RESOLVED )
8:09 am

Connectivity issues in one of our datacenters ( RESOLVED )

Posted (January 30th, 2012 at 8:09 am PST) by Justin@DH

As of 8am PST, we have been experienced connectivity issues in our Downtown Los Angeles datacenter. The other datacenters aren’t affected. Our system administration and datacenter operation team was engaged right away and promptly restored some of the services. Our network engineers are also looking into the problem, and we are continuing to fix up remaining issues and investigate possible causes.

Update Jan 30th, 10:15am PST: Our sincere apologies for the intermittent network outage this morning especially coming on the heels of the downtime experienced yesterday. All services have been restored.

We’ve investigated the cause of the network outage and determined that a distributed denial of service (DDOS) attack severely impaired a network link between our border and core routers. We have DDOS mitigation systems in place, but given the increase in activity in 2012, we’d already been testing new more powerful systems to implement within the next month. We’ll be accelerating this implementation.

We will be scheduling an 8pm PST emergency network maintenance window tonight to make some network configuration changes that will mitigate against DDOS attacks in the short term.

Please contact support for any further assistance needed. We will be creating a separate status post for the maintenance window tonight. – Ralph

Update Jan 30th, 9:15am PST:We are still collecting data. We will have more details for you a bit later. Services are mostly restored, although the control panel is still being worked on. We hope to have full services restored shortly. – Justin

Update Jan 30th, 8:55am PST:We are starting to see some services coming back up, and our admins are continuing to work on any remaining issues. The DreamHost control panel for example still appears to be having issues and is being looked into. – Justin

Update Jan 30th, 8:40am PST:Investigation is continuing, as our admins check all the links to the datacenter and other possible causes. We are doing all we can to get this resolved as soon as possible. – Justin

Update Jan 30th, 8:20am PST:All personnel are notified and are rushing into position to investigate and get this issue resolved as fast as humanly possible. We do not have any new info at this time about the cause but it is being investigated. – Justin

This entry was posted on Monday, January 30th, 2012 at 8:09 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

553 Responses to “Connectivity issues in one of our datacenters ( RESOLVED )”

NOT AGAIN. UGH. I’d just recovered from yesterday.

you are killing me.

again? wtf?

Seriously? Again? Didn’t we just do this song and dance yesterday?! Absolutely ludicrous.

Thanks for the heads up! Let us know if there’s anything we should be doing!

again?

January 30th, 2012 at 8:12 amSuper Frustrated Says:

Amateurs.

this is insane, i am tired of dealing with this downtime every day, getting very fed up

AGAINNNNNNNNNNNNNNN ?

Again down?!! Nooooooo!!!

Every week it seems like there is a problem with Dreamhost now. I been a customer since 2010 and Lord knows it my ass is on the line and this starts happening. What is going on over there! Why are there so many problems with just functioning!

I cannot even access the panel.

I really thing you guys should remove the 100% guarantee from your website

bring it back! no imap, no webmail, no everything! people are dying!

Enough already!
I for one would be willing to pay 20% more if you purchased more reliable equipment and hired better administrators!

Since January 20th our sites have only been for 3 days, and even then they werent working at 100%. Argh!

Not good. Lots of work to do on this Monday morning, can’t get it done.

: (

I had planned to upgrade to a dedicated server this week, but the significant troubles in January are making me wonder whether that would be a bad decision. I’ve been with DreamHost for many years. I’d like to stay. I hope you fix things in a more permanent way.

ಠ_ಠ

A lot of people already saying “Again?” But my question is “Still?!”

I hope you all can get this under wraps soon… good luck! But this has been a horrendous month for Dreamhost customers (and no doubt the staff too).

Hope things stabilize soon.

Un. Be. Lieveable.

I’ve got angry clients here.

Been with DH since 2006. Really, really sad to see how far things have gone downhill in recent months. Downtime, security issues, everything has gotten worse just in the last few months.

thank god at least this status page is available…

Where is dreamhoststatus.com hosted? It always seems to be up!

Guys, seriously?!?!

I know sh*t happens, but c’mon! This is getting ridiculous!

What the hell? 8am? You’re just noticing this now?

nahhhhhhhhhhhhh are you kidding me ? againn ?

These datacenters are located in Lybia, Marrocos and Colombia ? Damm

Again…………

They were up long enough to send me their monthly bill. Now they’re down again.

I’ve had opportunities to recommend DH to friends but haven’t because of these issues. They are coming far too frequently. I understand that “things happen” but this is insane.

yeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeesh!!!!

any way this is a DDoS or anonymous related?

Frustrating, but I know that every machine breaks down sometimes. Best of luck getting things fixed. I will bounce my dns over to an AWS instance for the outage and come back as soon as you guys post that things are good. Thanks for the notification.

“DreamHost STATUS Whats Up”… “DreamHost STATUS Whats DOWN” more like.

This really is unbelievable.

January 30th, 2012 at 8:14 amanother frustrated customer Says:

been with dreamhost for a LONG time – and this past month or two has been painful. downtimes, security breach, resetting every domain to resolve hack as a result. brutal

I hate to say it — I’ve been a customer for — not sure, 7 years now? And though I’ve held tight, I’ve brought on — I can count 6 clients from the past year alone – onto dreamhost; I’m going to have to think about recommending you anymore as I’m taking a hit, hearing MY clients yell at ME.

This is getting to be ridiculous. We have a VPS set up to run our business. Right now our business is dead in the water. What sort of reparations is DreamHost going to be making to their customers? I guess my day today is going to be spent researching other providers and moving our business elsewhere. We can not continue to operate on consecutive days with this downtime.

Anyone have any ideias of the causes?
Hackers? Hardware? Energy failures? Bug? Admin erros? WTF!!!

Everyone is losing money every day with this dowtimes…
Maybe it would be a good ideia pay more anyhwere else for a good service.

Any ideias?

jesus christ

fix your CISCO NETWORK!

Perhaps after the mass exodus of customers away from Dreamhost to more competent providers, their network will not be so overtaxed.

Panel dreamhost: no

Webmail: No

Sities: Yes

Fixing: Idont know

Please Helpppppp

how to can help me

They can’t keep the site up long enough for me to move it.

I want my site back up. I just got published on another site and will be losing significant traffic!!! This is not good for business.

DreamHost is rapidly becoming NightmareHost!!!
Enough already. I have been a long time customer and I am tired of these repeated crashes of your entire system. I will start looking for a new web host this week. How do you expect to keep customers if this happens every week???

So my blog is back but now no email … I am looking to switch to a reliable provider

guys, i’ve been a customer for 11 years, but at this point i have to seriously consider moving off. this is a huge issue, impacting my business.

does anyone know of a different host company that is reasonable but more reliable? I can’t run a business like this!

it seems your shmustomers are really shmangry.

I’ve been hosting with Dreamhost for years now and I’m about to jump ship. This is ridiculous.

Client since 2002. What a terrible month for Dreamhost. Not only will you be losing clients, but the consultants that recommended you are, too.
Bad stuff…

How do you stay in business? My website was down for about a week from the last outage. I’m told I should switch to Google mail… maybe I should listen to my friends more often. WHAT A JOKE

This is absolutely dreadful. Dreamhost, you’re totally blowing it. I can’t believe an organization of your size, and expertise, would let something like this happen, on such a huge scale, two days in a row. You’re hurting the businesses that you host.

Pretty sure that http://www.dreamhoststatus.com is hosted on Linode.

Are comments being deleted? I know there were some from 5am 10 minutes ago, and now they’re gone, as is my comment from 8 minutes ago.

Thanks for letting us know. I appreciate you getting right on it.

We don’t run our business on Dreamhost for that reason. We just use it for some inernal-use wikis and the like. So it’s annoying if it’s down, but we don’t lose any customers or money. Our customers are on a high-end hybrid cloud+colo environment and we pay $3,000/month for that, but have never had any downtime ever.

Down again?! Are you kidding me?!
Che, aparte, se nota que un montón somos hispanohablantes, el spanglish salta a la vista, jeje.

If I had hosting as reliable as dreamhoststatus.com we wouldn’t need dreamhoststatus.com

Two days in a row. We’ve lost real $$$ and trust with our users.

Get it fixed, give us a credit or we are gone and for good.

I wonder how old all this hardware is that’s been failing recently.

Did some one not payed the energy bill? WTF???? AGAIN!

I don’t think anyone else here that is concerned about their sites, and the status of this problem, cares about how long anyone has been a customer. I want to see important updates, not your whining. Bring it up in your personal channel to support, not here. Please and thank you.

Come on! Seriously?

Something has drastically changed in recent months. I’m sure (?) Dreamhost is analyzing just what the core issues are and getting back to the reliable and “Pro grade” service we are all used to. I put my name in association everytime I recomment Dreamhost to a new client, so … we are ALL in this together.

This has gotten a touch ridiculous.

This is not fun. It is starting to get tiring. Please fix this once and for all.

Get this fixed, you are killing me, I’m in the middle of a huge project and need e-mail ASAP!!!!!

Even dreamhost.com is down.

servicio de mierda que tienen

Nevermind, didn’t realize this was a new post :P

omg all of my websites are down. this is fucking ridiculous. someone PLEASE post a how-to on moving hosts

Not cool. I too have been a loyal customer for nearly 8 years with no issues till lately. It is frustrating and embarrassing to explain to clients why their site/email aren’t working. I am shopping for another provider and only hope DH can clean up their act. This is unacceptable.

I earn 1 USD per day on a Google ad. That feeds me with a one Dollar Mac. Now I have to go diet.

Can you please just leave it working for one day so I can download all my sites and leave?

At $9.99/month you are paying 33 cents a day. That’s just over 1 cent per hour. Have fun collecting a refund for downtime.

5th Major outage this month!

Ugh. Losing money. Not just immediately but who knows what the long term trickle down effects will be. Not good. Been with dreamhost for about a decade now… but this is beginning to hurt.

KNOWNHOST VPS:
1200MB RAM for $45/mo ($39/mo every mont with 15% coupon)
960MB RAM only $35/mo ( $29/mo every month with 15% coupon)
1472MB RAM $65/mo
1600MB RAM $85/mo
Been with them 4 months and never down.
Cpanel is $5/mo extra but with 15% discount coupn its practically free.

Ugh, I picked a bad day to go live!

You guys are costing me business right now. I sent off a very important email right before servers went down, now there’s NO WAY for me to get the reply!

THIS IS UNACCEPTABLE, and I am very angry. If you don’t plan on compensating your customers in some way, then I will take my business elsewhere in the future.

Jeeze people. Settle down. I’m sure it will be fixed promptly.

6 year VPS customer.
You do not have 99.9% uptime.

Customers should get a full refund this month.
If not, I will be leaving in April… 100% guaranteed.

Dear Simon Anderson,

Who is getting fired for the outages for 3 straight days so these problems do not continue to occur?

-Mike
NEF

Ok, Dreamhost you have clearly demonstrated you are unable and incapable of running a hosted service. We have been a VPS customer for 2 years. You should fire some people on your end. I will fire you on my end.

Yeah it sucks but… I have been w/ DH for 4 years and I have had next to no downtime ever. I would recommend DH without reservation. Just my experience…

DOWN AS WELL WTF!@

Oh great, what cable have they tripped over now?

Please make haste in resolving this, gentleman. It is Monday and there is work to be done. All our users are dead in the water.

i’m outta here. /longtime disgruntled customer

loligagon. is good that 3ezy.net uses multihomed hosts. Only one point is on DH. 3ezy.net is forevaaaa

all my shared nad VPS websies are dwon

You seriously need to get your shit together. This is two days I’m down now. Will you be reimbursing me for my lost sales?!?!?

Alguien conoce una alternativa? en un mes me toca renovar…

Anyone know an alternative? in a month I have to renew … (by Google Translate)

Thanks Dreamhost for letting us know. Hopefully the problem will be solved soon.

My client’s VPS is still functional and I can still login to our shared server and VPS. However, the control panel is down at http://panel.dreamhost.com as well as the TLD http://dreamhost.com.

Mondays are my client’s biggest traffic day so I’m hoping our VPS site stays live. I’ll be keeping an eye on it.

ΚΝΟWΝΗΟЅΤ VPS:
1200MB RΑΜ for $45/mо ($39/mo еvеrу mont wіth 15% cоupоn)
960ΜΒ RAM оnlу $35/mo ( $29/mo еvеrу month wіth 15% cоupоn)
1472ΜΒ RAM $65/mо
1600ΜΒ RAM $85/mо
Βееn with thеm 4 mоnthѕ and nеvеr down.
Cpanel іѕ $5/mo еxtrа but wіth 15% dіѕcоunt coupn іtѕ practically frее.

KNOWNHOST VΡЅ:
1200ΜΒ RΑΜ for $45/mо ($39/mо every mоnt wіth 15% cоupоn)
960ΜΒ RΑΜ only $35/mо ( $29/mo еvеrу mоnth with 15% cоupоn)
1472ΜΒ RAM $65/mо
1600ΜΒ RΑΜ $85/mo
Been wіth thеm 4 mоnthѕ аnd never dоwn.
Сpаnеl іѕ $5/mo еxtrа but with 15% dіѕcоunt coupn іtѕ prаctіcаllу free.

KNOWNHOST VPS:
1200MB RAM fоr $45/mo ($39/mo every mont wіth 15% coupon)
960MB RAM оnlу $35/mo ( $29/mo every month wіth 15% coupon)
1472MB RAM $65/mo
1600MB RAM $85/mo
Been wіth thеm 4 months аnd never down.
Cpanel іѕ $5/mo extra but wіth 15% discount coupn іtѕ practically free.

It is impossible for us to conduct business without the internet. Our customers book talent directly via email. Who knows what we have lost in the last two hours alone? It’s irresponsible for Dreamhost not to have a contact number, other than a fax. It’s also irresponsible not to send notification to those of us who help you stay in business.

@squirrel

That’s what WhoIsHostingThis says. Very ironic Drreamhost Status is not hosted by Dreamhost, but instead Linode.

All,

What can we do here? Been with dreamhost for over 10 years. This is crap. I can’t send / recieve email. I can’t tackle my to-do list. Loosing valuable billing time!!!!!!!!!

GET IT WORKING PLEASE I AM LOSING MONEY.

Can anyone suggest a better host? I want to switch.

Best,

John

“At $9.99/month you are paying 33 cents a day. That’s just over 1 cent per hour. Have fun collecting a refund for downtime.”

Speak for yourself… I’m paying $50 a month because I thought a VPS would be more reliable. But instead, I’m having more down time than ever before. Had a big post to do this morning and still can’t post it. Now I’m losing cash and visitors.

I’ve been a customer since 2006 and I’m starting to get really fed up with the service (or lack of) over the last few months. I do appreciate that DH has really responsive Customer Service, but that doesn’t help when it our sites are down on a constant basis… or so it seems. I’m bleeding traffic here DH… Please HELP!

This is a company going down the shitter. glug glug glug

out of 30 sites ibacked up 1 last night now what ?

Who´s gonna refound me for each client I lose?

I can’t work now … do you know that I’m supposed to be paid to work with dh interface? I’m using dh for my job … so bring back the service up quickly it’s urgent!

You have demostrated a complete lack of professionalism and responsability. I’m sorry, but I wouldn’t recommend Dreamhost to anyone. And I’m probably going to migrate myself.

NO SITES/NO EMAILS/NO FTP/NO PANEL – IS DREAMHOST IMPLODING?!

TWO DAYS WITHOUT BEING ABLE TO TAKE A SINGLE PENNY ON OUR WEB SITE, NIGHTMAREHOST WILL SURELY COMPENSATE I HOPE???!!!

Dh is notloosing any money, they still charging us as we speak

Not again!!! Every week is the same…. every monday… this and the previous issue with the security only makes me think that DH is slowly disappearing.

At least I can use this downtime to finally start setting up my sites on Amazon EC2.

Seriousally !!!

is this a joke???
you guys are pathethic…
stinky..

you already cost me over $2000 loss, in last few minutes.

YOU ARE BAD DREAM HOST..
YOU ARE NIGHTMARE HOST…

ALL EMAILS ARE DOWN, ECOMMERCE SITES ARE DOWN, WHY YOU DONT SHUT IT DOWN ONCE FOREVER. AND MAKE YOURSELF FREE FROM THIS MISERABLE LIFE OF MANAGING FEW SERVRERS. AND GO TO SOME WHERE AND SELL BURGERS.

Some of us pay a hell of a lot more than $9.99/mo.

I’ve been arguing with The Man to ditch DreamHost for months now. I think this outage sealed the deal! Woohoo!

I agree.. this is out of control and happening way to often. Where is my credit to my account for the inconvenience and the 100% guarantee!!!!????

Can someone recommend a better host?

This is really frustrating. Over 3 year customer. Now its costing me money with sites down and now the panel down as well.

And this is what happens when you oversell so hard like you guys do. I’m not mad or upset that there is downtime. i paid 10 for a year of hosting. To expect it to be good or reliable would be ignorant on my part. I refuse to host anything thats critical in regards to uptime at DH. And why everyone is all upset and surprised that dreamhost isn’t all that reliable is just a hosting noob.

Stop overselling your resources, stop promising resources you can’t realistically allocate to each user. Your problems will go away.

@Rob

Agreed – I’ve also been with them for 6 years and this is the worst I’ve ever seen.

this board is the only one thingthats not down, all the rest ofthe DH is fu**cked up

AHHHHH geeez, why does this ALWAYS happen? I am going to have to point clients to other hosts…I am sick and tired of them ranting and raving every two seconds and me taking the blame!! We all know hosts have to deal with tons of problems, but if it’s not one thing it’s another….just recently their database is broken into?

today is a mission critical day…have to set up emails and modify sites and I can’t do it!

January 30th, 2012 at 8:28 amDaniel Cáceres Says:

DH 2 days with A LOT of PROBLEMS. Who respond for our interest & monney?

NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!

When will the outages be resolved??

Did someone forget to “wr mem” on a switch again? Reboots are a bitch when you come back with factory settings.

i bet dreamhoststatus is hosted on EC2, lol. I sure moved a lot of stuff there.

YOU KNOW YOU GUYS ARE LOOSERS ….
YOU AR PATHETHIC…

apurence mierda!!!!!!!!!!!!!

Help my vps down !

Urgent !!!

again? D: 2 days after a show when i’m expecting customers to visit my site…! arrrgggh!

Get a grip people! How much are you paying annually for your hosting? That’s what I thought. If you want 100% uptime on all of your projects, tell me when you find someone that can actually deliver. My DH account is a playground. My company pays a few thousand dollars a month for a dedicated private cluster that is managed 24/7/365. My point is, YOU GET WHAT YOU PAY FOR! DO not expect 100% uptime in this kind of environment.

People, if you’re running a serious business and need constant uptime, you shouldn’t be with working with this company. This is what, their fifth outage in a month? And the second in two days? Yesterday was the last straw for me. Now I am just rushing a move out of here.

http://www.dreamhoststatus.com is hosted by Linode. Maybe if we all switch we can get a group discount :)

KNOWNHOST VPS:
1200MB RAM fоr $45/mo
960MB RAM оnlу $35/mo
1472MB RAM $65/mo
1600MB RAM $85/mo
Been wіth thеm 4 months аnd never down.

i’M VERY upset ABOUT dREAMHOST… tHE LAST MONTHS MAKE ME LOOSE A LOT OF TIME & SEEM A BAD PROFESSIONAL WITH MY CUSTOMERS….

FUCK YOU DREAMHOST! THIS IS ALREADY BELOW THE BELT! CC: SIMON ANDERSON.. DO SOMETHING ABOUT THIS OR FUCK YOU TOO! AND YOU EVEN HAVE THE FACE TO SEND THE BILLING FOR THE MONTH OF JANUARY. FUCK THAT

HELLO GUYS

I WAN TO MOVE AWAY FROM THIS PATHETHIC NIGHTMARE HOST. PLEASE RECOMMEND ME SOME GOOD HOSTING COMPANY.

This is an absolute joke. My clients are going bezerk. You are losing everyone business and damaging OUR reputation in the process. I’ve been a client for many years but i’m now looking elsewhere. My developers have said for years that i should move but because of the support i receive i have stayed with you. I’m afraid that is it now. i can’t even access the panel. It’s a disgrace.

This is ridiculous….And i just paid for another year. There is no doubt in my mind that something is seriously screwed up over @dreamhost. I am moving all of my sites ASAP. Let the word out folks, tell everyone how bad dreamhost has become!!!

KNOWNHOST VPS:
1200MB RAM fоr $45/mo
960MB RAM оnlу $35/mo
1472MB RAM $65/mo
1600MB RAM $85/mo
Been wіth thеm 4 months аnd never down.

@squirrel, 3ezy.net uses DH , as well as Linode and auto failover to a 3rd site. that costs but when people scream business critical, this is what we need.

Hola, Estoy molesto con todos estos problemas! ayer estuvieron TODAS mis paginas caidas por un problema de memoria y hoy pasa esto…

Estos perdiendo dinero, me estan ocacionando problemas con mis clientes esto ya no puede pasar mas! yo soy un cliente de Dreamhost que hace todos sus pagos al día y pensaba seguir asi por mucho mucho tiempo.

Hoy me puse la tarea de buscar otro proveedor de alojamiento web.

********************************************************************************

Hello, I’m upset with all these problems! yesterday all my pages were dropped by a memory problem and now this happens …

These losing money, are occasions me problems with my clients that can not spend more! I am a customer of Dreamhost to make all your payments on time and planned to continue so for a long long time.

Today I got the task of finding another web host.

I don’t really want to look for a new host, but this down time is unbearable.

January 30th, 2012 at 8:31 amIrritated Customer Says:

This is becoming a habit. I have always loved the service but this is getting unacceptable. You guys suck at life right now.

ΚΝΟWΝΗΟЅΤ VPS:
1200MB RΑΜ fоr $45/mо
960ΜΒ RAM оnlу $35/mo
1472MB RΑΜ $65/mo
1600MB RΑΜ $85/mo
Been wіth thеm 4 months аnd never dоwn.

I WILL BE FUCKED BY MY CUSTOMERS WHOM I AM HOSTING.

YOU GUYS ARE LOOSERS..

SHAME ON YOU..

STOP MANAGING SERVERS AND SELL PIZZA

I M SERIOUS

What the…. I have my yearly fee due in a couple of weeks I’ve have been a customer since 04 and now I’m seriously thinking of not renewing and going elsewhere. I was considering a dedicated server too.. sheesh! To put it mildly.

Oh if anyone is looking for some other hosting check out hostgator, they’re affordable and better than DreamHost. Best of all they use cPanel (woot) http://bit.ly/b7cT4W

On the bright side, even with this downtime…DREAMHOST IS STILL BETTER THAN GODADDY! :)

BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!BASTA !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Me too want to switch. If anyone know any better host, please suggest. Thank you.

I give up. I have clients yelling, one high end client threatening a law suit for loss of income. Told all of them I would have their sites moved off this morning and now…. MORE YELLING!!!!

This downtime will only cost me a few hundred dollars, but to me, that’s money I don’t have to lose :(

So Dreamhost tried to tell me my website was slow and down because it was using “More than it’s share of memory”… they told me to upgrade… are you sure Dreamhost??? I’m thinking maybe it’s YOU :( I refuse to upgrade, pay a lot more a month, just to be down all the time. This looks very bad for our website.

Why? Hagan algo por favor…

thanx for the update!!

ignore all the whiners….just get it fixed please….

I love all the affiliate hosting vultures coming out as soon as they smell death.

KNOWNHOST VΡЅ:
1200ΜΒ RΑΜ fоr $45/mо
960ΜΒ RΑΜ оnlу $35/mо
1472ΜΒ RΑΜ $65/mo
1600MB RΑΜ $85/mо
Βееn wіth thеm 4 months аnd nеvеr down.

Id like to know where this status page is hosted! obviously not with Dreamhost.

Help my vps down !
Urgent !!!

I totally agree. 100% uptime is a crock of s***.

try to be more Professional !!
Damn !!

squirrel Says:
http://www.dreamhoststatus.com is hosted by Linode. Maybe if we all switch we can get a group discount

WHAT A PATHETHIC COMPANY , THAT THEY HOSTING THIER STATUS WITH ANOTHER COMPANY SURE ALL MOVE TO OTHER COMPANIES.

Well, that’s it. I’m moving all my sites this week. I’ve been a customer for 3 years, brought many clients there and because of it I look like an idiot.

Sad, I’ll miss the admin panel. But damn, even GoDaddy offers better performance and reliability.

DH down again and even unable to login to panel.dreamhost.com I think it is time to start looking for another host and registrar. This is crazy.

THIS IS BS! You guys need to get your shit together.

DAMMIT! NOW SITES DOWN!! Just after a social media blitz too nonetheless…ugggh..all links pointing to a dead fricken server!!! That’s it…I think I’ve had enough…

at least they are nice enough to give us a place to rant…a nice little corner of the web where they don’t have to listen to us :(

You folks need to stop whining. Do you think that hurling abuse at the staff is going to make them work any faster than they already are? OK, fair enough, you pay your money to expect a certian level of service…but no-one is forcing you to stay!

I have been a customer for over 8 years and in that time I have experienced a few downtimes and have been frustrated with things but DH have always done their best to get the service up as soon as possible.

Give them a break! If you are not happy. Move.

Waving your hands up and down is immature and pointless.

At least their offsite DNS is still working. Otherwise I’d be losing a lot of email right now. (This is of course a big part of why I do my own email hosting on a separate VPS that isn’t hosted by Dreamhost.)

Neo, they host with another company to prevent a common source of failure. If there was a major network outage, like right now and this status page was hosted at DreamHost, we wouldn’t be able to see the status page. This is an industry standard.

NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!NOT AGAIN !!! PLEASE FIX THIS ASAP !!!

holon i’m a page five of the manual .. should have evry thing fixed soon ..
meanwhile:
http://www.youtube.com/watch?v=EoOrtvYTKeE&feature=player_embedded

Again?! what’s wrong with you guys, I’m losing a lot of money for this

@John

Believe it or not linode.com sounds engaging. after all they’re hosting dreamhoststatus.com.

Good Afternoon,

All our hosts are down. What is happening?

We need urgent answers.

Sincerely,

Tatyana Sousa.

http://www.knownhost.com/vps_packages.html
1200MB RAMom meomory for $45/mo
Been with them since September last year, and never down.
Customer support response under 2 hours

Can I bill you guys for my lost revenue today? Many of us will loose more revenue from this outage than a year’s worth of hosting payments! Time to move on I guess.

What do you mean, they are rushing into position??? THEY SHOULD BE IN POSITION ALREADY. Right???

>:(

This is very very far from being okay. I host a lot of websites on Dreamhost, this downtime is not acceptable to me or my clients.

http://www.knownhost.com/vps_packages.html
1200MB RAMom meomory fоr $45/mo
Been wіth thеm since September last year, аnd never down.
Customer support response under 2 hours

God speed on finding a cause and solution. On another note, you should probably look into hiring better (or more) network engineers/sysadmins.

http://www.knоwnhоѕt.cоm/vpѕ_pаckаgеѕ.html
1200ΜΒ RAMom mеоmоrу for $45/mо
Been wіth them ѕіncе September lаѕt year, аnd never dоwn.
Сuѕtоmеr support rеѕpоnѕе under 2 hours

Sooooooooooooooo. Do we get any kind of compensation for downtime or lost revenue?

Dreamhost you’re costing me money, and all these issues lately i think it’s time to move to a better hosting provider. This is the LAST time i use Dreamhost!

For the last 5 days there has always been some issue. Get your shit together!

Does DH keep getting hacked? Or is old hardware dying? it’s one thing after another.

hey quit bitching about your biz critical apps. if its critical pay for it in $$$. We are paying 15,000$ per month for some of the client sites and we still get some (few minutes) downtime every year. DH is good for personal, small biz stuff. If you’re making money gtfo and spend some.

retards http://checkmysitestatus.appspot.com/

help all of my website I have with you is down

Greetings
Stefan Meller
HeeWeb.dk

http://www.knownhost.com/vps_packages.html
1200MB RΑΜоm mеоmоrу for $45/mо
Βееn with thеm ѕіncе September lаѕt уеаr, and nеvеr dоwn.
Сuѕtоmеr support rеѕpоnѕе undеr 2 hоurѕ

http://www.knownhost.com/vps_packages.html
1200MB RΑΜоm mеоmоrу for $45/mо

ya llevan 30 min!!!

You should hire some technicians for telework from Europe, where the 8am PST are the 5.00pm CET and the 3am PST are the 12pm CET, normal work hours. It won’t be the solutions if *any* datacenter will be accessible, of course, but it could be a good start to work on the problem without the need to wait for people get to their workplace. In case you’re interested, consider this as my application ;-)

You guys are killing me!! I can’t wait to be back up and running so I can switch away. You guys are awful!!

this is unacceptable and fortunately not something that we see regularly with Dreamhost speaking as a customer of 10 or more years. I cannot even access my control panel let alone any of my websites on the affected server, and of course, zero email. Please spend some money on equipment or personnel. Thanks.

Linode has VPS plans starting at $20/month. Maybe with a group discount we can get it down to $10 :)

Ugh… so that is why my business has been very quiet since yesterday!! I can’t believe my website hasn’t been back online. I host all my product photos on DH and they’re all broken!! :(

Paul, come on that’s very noble of you, but we aren’t talking about an occasional downtime here are we? And thanks…I think a lot of us are gonna take your advice.

waving your hands up and down is immature and pointless? in this economy where you got to put food on the table you expect composure? I am sure I would agree with you if, as I said, it was an occasional nuisance.

Come on DreamHost sort it OUT !

Time to move on…the writing’s on the wall

http://www.knownhost.com/vps_packages.html
960MB RAM only $35/mo ( $29/mo every month with 15% coupon)

Whoever is giving you the PR advice of never posting an estimated timeframe is doing you a disservice. It only adds to the angst of your already frustrated customers. We don’t need 27 updates of “we’re working on it”…of course you’re working on it. Instead, provide information that doesn’t keep us checking on the status all day.

wow……….. just wow..

took the money and ran..

Stop whining and take the punishment you paid for. Or that is what the DHfanidiots love to say. I missed my window to move off this host since my DNS is tied to my renewal and can’t be changed within 30 days of renewal.

I’m with Tony above. Fuck You DreamHost, and Fuck you Simon Anderson. You sure fucked me. I’m sure you are laughing your ass off with all these people’s money as you send out your fucking smarmy newsletters and updates about how shiny Dreamhost is when you know you run this shit in a closet of TRS80’s you bought at a garage sale. You keep referring to a “Datacenter”. Is that Datacenter in an old abandoned school bus or the tree house you built for your 5 year old? 6 years I’ve been doing this crap because I keep missing my window of opportunity to move. I’m monitoring it closely this time. Call me gone and the DHfanidiots can rejoice in my absence.

fboutique , host photos on a managed host like images.gonzb.com (paid) or 3ezy.net (free). It’s far better than dreamhost or relying on it if all you do is host pictures. imgur isnt bad either. all are in 5$/month range.

http://www.knownhost.com/vps_packages.html
VPS ,960MB RAM only $35/mo ( $29/mo every month with 15% coupon)

I’ve been with DreamHost since 2007 ad continuously experienced better service than with any other webhost I’ve tried. All my websites at DreamHost are down at this moment, but c’mon guys, get some perspective.

If your website is worth “few hundreds bucks per hour” have it mirrored on multiple webhosts.

Okay everyone, I’ve had enough! It’s Monday morning on the east coast and once again I can’t get to my email!

These folks have been having one problem after another for the last few weeks.

It’s clearly incompetence! I am losing business and can’t contact my customers. Unacceptable!

I think we should all get together and file a class action lawsuit. OR everyone leaves dream-host!

I’m calling my lawyer now, anyone on board with me? Contact me at paul@massproductions.net

January 30th, 2012 at 8:40 amIrritated Customer Says:

Rick Says

“Can I bill you guys for my lost revenue today? Many of us will loose more revenue from this outage than a year’s worth of hosting payments! Time to move on I guess.”

Amen. Nothing like a morning check up that makes you realize you are paying for advertising in areas where nothing exists.

Thank you @Dreamhost may I have another?

I have a bounch of clients yelling in my ears cuz they dont have mails

I believe this is what they call adding insult to injury.

I dont know what are you doing now?

More frequent updates, please.

This has become unacceptable, as is the problem of no phone support. We are shopping for a new host.

Thank you for providing a lot of good tech support and value for very little money. It seems though that there was some downtime over the weekend, too? We feel a bit anxious sometimes as this has happened a couple times and our business has a number of sites on Dreamhost. Hopefully you’ll get this sorted out quickly. It would be good to know what caused the outage and if there is something that can be done to prevent it in the future.

10 websites down….Am screwed.

wow i picked a bad time to move 60 sites to dream host.
now i have to go through the dance all over again with another provider

If you’re making that much money from your customers, you make enough to spend more than $10 a month on hosting. Buy a better hosting solution for your business.

Paul, good luck obtaining Class Action status, or good luck even getting any form of litigation started. DreamHost isn’t responsible for downtime, and you both read and agreed to those terms when you signed up.

Yeah enough with the whining. I’ve been a DH customer for probably five years or so and this is the exception not the rule. Take a deep breath. Give it a break. I’m sure things will be back up soon.

far too often. ugh.

January 30th, 2012 at 8:41 ammarcelo araujo Says:

all my sites are down! what a motherf… is this???? ASAP repair this! im losting a lot of host clients!

Our trust on dreamhost is going down ……………

Please be honest with us about why these outages are occuring. It will help us to have confidence that you are addressing the issue properly to insure stability in the future.

Thanks!

Chilllll…. paul.

Looks like it may be inside comcast or cogent (depending on which direction I do the traceroute)

January 30th, 2012 at 8:42 amChristopher Bright Says:

I’m having a feeling that they’re just meeting a bad time. Hardware all failing at the same time, etc.

By the looks of it, this is just a really bad coincidence.

Considering this is the only real downtime I have ever got under my 3 years with Dreamhost, I still have my full faith that they will fix things.

-Chris

It makes sense that Dreamhost would host their status site with another company. If dreamhoststatus.com was hosted by them we would have no idea what was going on. I think it is a smart move.

http://www.fatcow.com/

I used to use Fat Cow – really cheap and almost no downtime. THinking of going back. I’ve had so many email outages lately I’ve been blaming my computer or my ISP, thinking “there’s no way dreamhost would be down this much”.

@Paul and other dreamhost defenders, your all a bunch of tools. I take it you all have no income personal pages of your cats. The rest of us have high end clients that are losing 100’s and sometime 1000’s of dollars a day.

Guys – we need money back

click my ref code for linode http://www.linode.com/?r=0ca24dc2e9a2b21cba8053674e4b570cae8fed7b
Been with them for an year. 30 minutes downtime. support is exceptional. Most admins recommend it over at serverfault.

BLOODY HELL GUYS!

13 pos-0-11-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.87.37) 87.968 ms 88.026 ms 88.039 ms
14 * * *

and

18 te7-1.mag01.lax01.atlas.cogentco.com (154.54.28.142) 86.644 ms te8-1.mag01.lax01.atlas.cogentco.com (154.54.28.146) 92.218 ms 86.283 ms
19 * * *

This is probably the 3-4th time in the last 2 months my VPS went down… Wish they’d prorate me for the downtime.

ಠ_ಠ

All the best with the fix guys… must be stressful.

Really sucks everything is down, so hopefully a quick fix.

oh no – not again – totaly error! WOW. I’m working since some years with dreamhost and sometimes the server fucked up. Its technic. But is there no quality control management? – is there no assistance to say – sorry – the next month is for free? – Nothing – this is absolute not funny and maybe it is a new generation who dont say anything to the clients. Its maybe the typical facebook generation “i like it not”. VIRTUAL is NOT real!!! Bad work and bad support. Your start was great some years ago but the daily work is ……

Again, WTF? This is bad for everyone. You will lose customers…

It seems to me that there needs to be more redundancy at the host’s level. Just like each of us, DH is going to lose lots of business over this issue, which could have been avoided if they had established a better infrastructure. So, while I am excited for my client’s sites to be back online, I am just as curious as to what DH plans to do to prevent issues like this in the future!

Wow, you pros out there don’t have an emergency backup hosting solution for your prized customers? Bummer. ‘guess you’re not as reliable as you like to sell yourselves. With the money you’ve been saving using Dreamhost, you should have been investing in disaster avoidance solutions. Either with your own hardware and bandwidth, or some cloud solution, or something. I probably lost money this month, but my clients have remained oblivious to MY hosting issues.

i used to use Dreamhost, then I took an arrow in my knee http://checkmysitestatus.appspot.com/

January 30th, 2012 at 8:43 ammarcelo araujo Says:

the DH site is down too! its a big shame! periculous things! im thinking seriously to get out of DH.

January 30th, 2012 at 8:44 amGreg DeMaderios Says:

I’m really curious as to why you would “autoupdate” any servers without going through proper testing and what not? This quote, “We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers.” just makes it seem like you’re asking for trouble, which you ran into….

I’m not trying to be an arm-chair quarterback. As a Systems Administrator, this type of setup makes the hair on the back of my neck crawl.

What a way to start the week ! even the dreamhost site itself is down making it hard to report outage – my site and email is totally down – second time this month – like many above this is simply out of control – for us on the East Coast this will be our entire work day gone up in smoke – not on… and as Hao says WHAT ARE YOU DOING NOW ? will this be fixed soon ?

This is soooo bad. ALl morning I have had the same issue with a clients calling me and asking what in the world is going on with their web sites. its not just webmail. its the whole darn site that is down.. all of them. This is not good. Downtime should be totally a minimum. I am with Paul above, We pay to much $ monthly on Dedicated Servers and VIP crap and this is happening way to often.
Not a happy customer and my customers are even more pissed off.

Everything I have is down;

websites

email

pants

confidence

Dream Sucked at Dream Host!

DAMMIT! Not F$#@# again!

the mother ship has landed:
te7-1.mag01.lax01.atlas.cogentco.com (154.54.28.142)
holom i’m a page six of manual

January 30th, 2012 at 8:44 amnytimesreporter Says:

This series of outages are really hurting our business, especially on a monday morning when reports are due.
Perhaps time to switch.

For me it’s just bad timing. 90% of our business is done on Monday, Wednesday and Friday mornings between 7am and 11am MST. If my site made hundreds every hour all the time, I probably would have money to spend on a better solution. Unfortunately it just hit at a particularly bad time for us. Oh well right.

This is so frustrating… Sabotaging sales campaigns as URLs are broken…

Please, just what is going on and how long before it’s resolved?

My Web site are up but really slow.

I guess I will be moving. Now I have to go back and explain AGAIN!!!! Within the same day to my clients!

I feel so sorry for all the techs at Dreamhost. This will be something to tell your grandchildren.

I agree with Maxx and that effed up idiot who says customers with 1000s of dollars a day? Spend money on better servers yourself or tell your clients the truth. *%$&ers

dreamhost should be embarassed and his director or CEO should apologize in person to all clients,

WE DONT NEED ANY REFUND OR COMPENSATION.

WE WANT SITES UP AND RUNNING.

NOW ITS HABBIT OF OF DREAMHOST. ITS REALLY PAINFUL.

IF DH MANAGEMENT HAVE A DROP OF PROFESSIONALISM LEFT THEN THEY MUST RETIRE AND START SELLING PIZZAS DOOR TO DOOR.

I THINK THIS IS WHAT THEY CAN DO , MAY BE , THEY ALSO DELAY PIZZA DELIVERY LOL.

IM MAD

ALL OF MY BUISNESS WEBSITES AND CLEINTS WEBSITES ARE DOWN.
AND I HAVE TO FACE ANGRY CLIENTS.

PLEASE GIVE ME SOME GOOD SHARED HOSTING I WILL MOVE.

@Andy
Tell me about it.
Thank God that I backed up the majority last nite.
Now I need the DNS settings.

Damnit!! i think the guy from megaupload went on a rage and sent all his minions to destroy dreamhost…

Whe should get atleast 1 month free from this :P

Justin baby,

the problem is with dns of dreamhost.com, check…

I like the little disclaimer on the side:
“If you are experiencing a problem that has not been reported here, check our web panel for more information.”

I can’t. No panel. No e-mail. No Dreamhost. I wonder why this blog still works?

Seriously people? Give them a chance to fix it… 11:09 eastern time, and it’s now 11:45. The people who work at DH like, have to get to work (where it’s not even 9 AM)

Been a DH customer for a very long time, and they are responsive, reliable and working hard I’m sure to fix this…

No one can predict when something is going to brake- cut them some slack!

I copied my clients in on the message given by the CEO this morning.

What is the point of lying to your customers?

Good luck on fixing this. Hopefully the problem will resolve soon.

Thank you for resolving the problem asap, my dear dreamhost.

I think there is a DNS issue – I notice some of my sites are back up, but those that are not are not resolving correctly.

Oh, and shut it with that $9.99 a month shit DHfanidiots. Lots of us pay a LOT more for DH than $9.99 a month and are getting $3.99 a month level of service.

Growing disappointed about DH…. last week change all passwords, now this…. come on!

Really? My sites seem to be up, but I have no panel access. I really like to be optimistic about DreamHost, but now I fear the worst is yet to come.

It would be fantastic if Dreamhost gave us all a referral credit or something for putting up with these issues.

The boilerplate “we are doing everything we can to resolve the issue as fast as possible” is not a status, it is a commercial–and meaningless, until there is an ETR.

I am with DH for years, but I learned my lesson that serious stuff ought to be hosted elsewhere with good backup as well. Click my name for the Good host to move to from DH if that’s what you are shouting about.

Thanks for the update! You’re a great company and I know this is tough. Keep at it, out of the many years I’ve been with DH, I know this isn’t normal so don’t feel bad! People just don’t get the intense frustration with network outages.

SCOTT

WHY THE f*** DNS ISSUE?

and tomorrow? What a surprise we have?

*points and laughs*

My two sites are up.

Also, hang in there dreamhost, sometimes crap happens. Don’t listen to these rude people. You all are doing your best.

Disappointed. Very, very, very disappointed. Back to Verio I think for me and the one domain I moved here from there. Verio – excellent 24/7 tech support, incredible uptime.

So disappointed in the quality here.

Email – no work!
Panel – no work!
FTP – yes work!
Site – yes work!

HELP! again.
please.

“jason Says:
i used to use Dreamhost, then I took an arrow in my knee”

Hah, well played sir, well played.

This is really bad..get it together please,…I lost my Google ranking & eventually was delisted because of security breaches & spam cloaking on my site. people are loosing money when this happens. With security breaches and webmail down I think it might be time to look for another web host. Sorry Dreamhost this is becoming a real nightmare!

I know you guys are very busy right now, but when you are done fixing your umpteenth failure, you can kiss my ass.

THIS IS NOT HOW YOU RUN A BUSINESS IN A COMPETITIVE MARKET YOU CUNTS!

Yo espero que no me cobren el dia de ayer ni hoy.

***********************************************

I hope not charge me the day yesterday or today.

Many of us appreciate all the great works you have done so far, a few hours of down time won’t kill us.

I started with DreamBooks in 1998 and then got hosting with DH in 2002 and I’m paid in full through 2017 because I’ve been a generally happy DH customer all these years. However, I’m sorry to say that service has declined since they hired a CEO. If there are other problems in the DH system that are causing this, then please address and resolve then as soon as possible. If not, or you can’t figure out what those problems are, then I suggest DH give some serious thought about what Simon the CEO’s leadership is doing to DH. Yes, DH was wacky, but it was a reliable, endearing wackiness. Sorry, but this is what I think. Maybe you need to take that big CEO salary and put it into more practical things, y’know, like hardware and maintenance. Just a thought from a longtime customer.

Also, even when I can get to the webpanel for it, I’m afraid to contact support for anything anymore. Support communication has gone to hell, too.

I am done with Dreamhost………I will never forget this.

Moving to a new host! You got a lot of people at my job pissed at me!

Mine is back up.

My server is giving problem please solve me .. are you always causing problems .. their service is pecimo .. I do not discount it more. I charged ..

Email down….

Thanks Dan, I just needed the right timing to say it. :D

E-mail just went live again.

w/ panel.dreamhost.com down we are all unable to transfer our domains to another host. Wonder how many people will leave once it’s back up. Where should we go?

R, I am as disappointed as everyone else, but do you honestly think screaming and shouting is making it any better? If any good can out of this it will help prevent these issues happening again.

DH are embaressed or at least they should be and as a result it should help put practises in place to ensure going forward is a more stable experience. We never know the full story because in most cases the powers that be don’t know themselves, but when they do they have been open about the problems and provide regular updates.

Not been a great week for DH but i’m sure this will result in more stability for the future.

11:49est My sites are back up

Is there some way to get dh to notify us when things are back?

January 30th, 2012 at 8:50 ammarcelo araujo Says:

its really a big problem!

Now they block negative feedback with profanity. just watch.

We seem to be up again. Hope it lasts.

Thanks for the help and updates!

seriously? what’s going on over there. this is becoming a regular occurrence. i have been an avid supporter of dreamhost, but not so much these days..

We just joined Dreamhost a week ago but this is making us rethink that decision since downtime was the very reason we switched from our previous host. We only knew about it because one of our staff members happened to visit the website over the weekend and let us know it was down. It would have been helpful if Dreamhost had alerted us that this was happening, maybe by text message if their email servers were down.

Thousands in business lost this morning-We will be missing airdates due to this problem. Not cool, especially having outages on a heavy business day (Monday morning).
Can anyone suggest a more reliable hosting company?.
thanks.

Yes, came back all access

I’m getting a partial site picture…Most of the text and some of the graphics are coming through but still no control panel.

SUICIDE! I Sell Super Bowl Packages so my window to do business is VERY VERY small. 48 hours of downtown is equivalent to 3 months for any other business. No refund will cover my loss. Time for a change.

Back up and running. Nice work. Thank you for the timely updates!

Attn: Simon Anderson and staff
EAT MY @SS!

looks like my sites are back up (were only down for a bit today). Thanks Dreamhost!

My sites are back up and the whole thing took less than an hour. This was a big hit and they got it pretty fast. I’m not in love with DH but I think they did pretty well here, considering.

Thank you for getting my sites are back up, but the panel is still down.

It is not a pleasant situation, but let’s go guys!
Solve the problem and make it possible for events like this do not happen anymore.
Thank you

My VPS is back up, things are finally looking good.

Websites up… panel still down.

Please what is happening? incidents are very serious and very straight, are affected the operations of my business and my clients. They affect all sites on the server my penguin, and cuadernosnebula.com alt24.net included

My sites on shared servers are back up. Panel is still down and I am able to get to web mail but cannot open any messages without getting yesterday’s error message.

Everybody take a deep breath. Life will go on.

This is frustrating, but it is not life or death. Go outside and breathe some fresh air.

Monday morning business via my website is critical. You guys are costing us money.

dreamhost.com came up a little while ago but now u are resetting connections, why ?

nevermind. we are up. you guys really need to tell us what’s been going on lately.. it’s starting to hurt production.

FTP still down…

Web panel (panel.dreamhost.com) is down!!!!!! and all my sites

Fred W – If you are making thousands of dollars every 45 minutes or so, you should probably be able to afford a better host.

@Noel – maybe if your sites were up you’d be singing a different tune! But you got yours so all is well no?

As far as “not listening to the rude people” – people that are responsible for their company’s hosting and domains are under a lot of fire right now. They are, after all, often the people who CHOSE Dreamhost. And people that are getting heat or losing money because of Dreamhost have a lot of adrenaline and would you rather they vent here on Dreamhoststatus.com or on the forums they frequent? Oh because these guys… yeah they frequent a bunch of tech forums – and there are lots of other options out there.

Mail is starting to filter back in.

my websites are back up too. And the panel ALMOST loaded, but it’s gone again…

SPEAK UP PLEASE we can only hear you in UPPER FCKN CASE!!!!

January 30th, 2012 at 8:53 amLong Time User (But...) Says:

I have been with dreamhost now for nearly 8 years. I have dozens of websites with them that are my own and my clients. This outage is going to cost a lot of people a lot of money and frustration. And the fact that this has happened twice in the last few months is unacceptable. Plus there seems to be more and more issues lately. But, they have provided years and years of excellent service. If we all pull out right now, it’s going to bring them down (and the rest of us). So if you CAN provide a little tolerance to them that would be great. I don’t know what my clients will make me do, but I am going to try to hold on to the “third time is the charm” rule.

I feel like somethings gone wrong with dreamhost. It’s not normal. I hope all is okay and I will try to keep my business with them to help them get fixed. If you are a business owner, and are able to, have some patience. You know that sometimes bad things happen.

There are tons of comments. Angry comments. Understandably angry comments. But hopefully this one can help solve the problem rather than just pass anger around the “room”.

We have 40 people depending on the email to do their job.

Back up and running, except for web panel.

Warning: mysql_connect() [function.mysql-connect]: Lost connection to MySQL server at ‘reading initial communication packet’, system error

lo que es el colmo es que ya me estan cobrando enero … x-(

lets make this one roll to 1000 commints , like we did yesterday for another major downtime.

Good time to move to a better HOST GUYS

CHECK THIS OUT, THIS IS THE BEST HOST YOU WANT TO MOVE TO
http://www.nouptime.com/

You can even test their support ticket right now

xD

Sites and mail is back up, FTP and panel still down for me as well.

very sad :-( please fix it permanently

Its back!!!! cry babies…..

Who hosts http://www.dreamhoststatus.com? I want that service.

Network Error: Connection failed

i just lost 36 billion in the 24 minutes the site was down. i want my $50 back.

I’m about to join the ranks of those jumping ship too.

Sites up. Panel down.

DH is online again :)

“Brian Says:

Who hosts http://www.dreamhoststatus.com? I want that service.”

Linode.

still just have
“Error establishing a database connection” when trying to go to my blog and just the word ‘Error’ when going to my gallery page :-/

I demand more beer.

OK sites are comming by

My sites are online again, but the panel of DH is down

My shared site just came back up. I’m with everybody else in being very upset about the downtimes lately.

back online again now. Cheers

Thank you! I appreciate you keeping us posted! :)

@ Dude – yeah, you’re right – nobody has been hurt by the downtime and nobody has any reason to be upset. We should all just say – well sorry, we’re not paying much so no problem.

These people are probably making up all the pain outages can cause.

Don’t tell me – you must work for Dreamhost with that mindset!

Definitely moving to HostGator when my next client pays, this has gotten way too far.

Looks like sites are back up, but databases are screwed?

Looks like I will be spending my time doing taxes, watching Netflix, and waiting. What’s Reddit hosted on? LOL

Camon! I thing it will be any kind of recompense!!
Again no, please! I’ve to WORK!!

i pusha green button:
xe-0-1-0.mpr1.lax7.us.above.net (64.125.30.5) 87.063 ms 89.276 ms 89.874
working now
i’m on a page 8 of manual
movin fast

Maybe when the dust settles 1 month of internet hosting credit is an order here for everyone!

I dont think its working. The beer tap on my site is not shooting any beers :( I want my 99$ back

Thanks. My sites are back running. Well done for the quick action (considering what time it is there).

No emails apologizing or explaining why they were down for so long yesterday.

They don’t have a problem emailing you when they’re asking for money.

I’ll say it again: DreamHost = The Zune of hosting companies.

Same here, an – getting “Error establishing a database connection” on our blogs.

I used to host with Hostgator like you, then I took an arrow in my knee.

We are having major email problems AGAIN! Can’t get to the Panel, can’t get to support. NO PHONE NUMBER FOR SUPPORT SERVICE! What a lousy way to operate. UGGH!

Good luck guys!

I’ve been an 8 year customer and this has never happened, everyone. This host as served me well and after this I will still be a customer for them when this is resolved. Every host has it’s problems from time to time, and for the most part DH has had their sites up 100% of the time. I’m sorry that newer customers don’t know this and I understand the frustration… My own sites are down too, but I know this will be resolved so everyone CHHHIIILLLLL.

Yay, thanks for the quick fix!

Jan 3: Ongoing networking outage
Jan 13: Network outage
Jan 18: Major network outage
Jan 29: Web, SSH, and FTP services down
Jan 30: Major Network Outage

My web site is down..
whats heppened?
Who can help me?

For the people losing thousands of dollar per hour – why don’t you pay for a High Availability configuration if your business is *that* critical? Seriously, a couple of separate servers (approx $10ea/mo for 256meg cloud servers on Rackspace for example) in different geographic-areas and a manual switch over for your DNS and it would be fairly cost effective.

You need to choose to your hosting based on what it is worth to you. If you hosting is only worth 33cents/day (for many domains) then just suck it up…

Well, this might be the last straw for dreamhost as I’m already on the hunt for a better PSP server.

They’ve got to be close to their 99.9% uptime (.1% = 8.76hrs) this year already.

mines all back to

hey those with database connection errors, just hang in there. seems like they got the webservers back online. its only matter of time to get the database server back. Same goes for email.

reboots take time ass**les stop cribbing, they seem to be on it.

Will we be refunded for sunday? I lost a couple hundred dollars :(

“Error establishing a database connection” for a client blog site

what’s heppened?

Dreamhost really sucks!

I will move to another Hosting!

If you have a “serious” website that makes hundreds or thousands of dollars a day – or even better, clients that have a “serious” website that makes hundreds or thousands of dollars a day and you’re leaving all that hanging on a service that charges $8.95/month you’re doing yourself and your clients a disservice and are displaying a serious lack of professionalism.

Grow a pair and talk to your clients about how hosting works and get them on a serious hosting package like linode.com or some other VPS or dedicated server. If you don’t know how to configure a service like linode.com, then hire a nerd to do it for you. Your clients deserve better.

I’ve been hosting with Dreamhost for years now and I’m about to cancel. This is ridiculous.

down again. seriously what about the 100% uptime guarantee?

move the server to this one!

Check out iPage.com. Ranked highest satisfaction of ALL domain hosting companies! You do not have to put up with old companies that are struggling to survive and refuse to put money back into their hosting! Come on over! iPage.com

“Quick fix”, “It’s back” and “Well done for the quick action”????? Are you guys frigging kidding me? This has been going on for over 24 hours for me. And I pay a premium price!

will we be refunded.. NO you won’t you are paying dinky shit dollars for a mediocre hosting package. get real hosting or stfu.

Bork! Bork! Bork!

I am on lione as well as dreamhost. which is why I lmfao here on the poor bastards who loose money and sleep over DH.

1. learn DNS load balancing. its 1 minute.
2. get Linode (click my name on this post)
3. Profit.

Did I mention only $3.50 per month for unlimited EVERYTHING!

Fuck you Wow.

Yup, all my sites and ftp are now up-and-running. It’s annoying, but I’m tempted to agree that if you are depending on your sites to make a living, Dreamhost isn’t – and has never been – the host for you. I can live with some downtime every now and then for the price – for important client sites that I really need to stay up, I host elsewhere.

I NEED THE WEBMAILS WORKING. EVERYTHING ELSE CAN WAIT, BUT EMAILS ARE TOO IMPORTANT FOR BUSINESS! PLEASE FIX THEM ASAP.

Thanks.

yes unlimited my ass. unlimited downtime. I really hope everyone moves out of dreamhost. I reallyneed the CPU time, what with all the idjits hosting ecommerce and making millions right next to my free site.

I have to put my cat picture on Google appengine , meh http://checkmysitestatus.appspot.com/

Yeah I think those of you who are freaking out and demanding one month free hosting and so on are hitting the roof over something minor. I am a web professional for a very large company and if your sites are that critical, then you need to take responsibility for your own sites and host them in a way that ensures they are up 100%. That’s not the host’s responsibility. No host is going to have perfect uptime. Obviously they will shoot for it but you need to have your own backup systems and what not. Your guys’ expectations are so out of proportion to what you are paying for it’s funny. I’m going to have to email some of my colleagues so they can have the good laugh I’ve had reading these comments.

Thanks Dreamhost! I now have more time to play COD and ignore yelling phone calls from clients.

yes maybe my answer to the email down shouldn’t have been said, cause i can’t log into my email now AGAIN!, haha umm.. my bad.

back up now, still switching though.

i’m sorry it was all my fault .. i added new admin user, took blue pill, and flipped yellow lever to off position by accident..
i refund everybody from my large offshore bank account soon
holon readin’ page nine of manual

if your sites are down , you can use this page http://checkmysitestatus.appspot.com/ in your dns redirection. it calms people down. except for cat haters or people with no sense of humor, we dont want them on the internet anyway. they are the ones who back SOPA >:(

Chill out people! smoke a joint. Kiss your wife. Make a cup of tea. Call your mother. Help someone with a task.
I wonder how many people will thank Dreamhost once the problem is fixed.

This is extremely frustrating. Last week one of our clients had their FTP passwords reset due to the compromise that occurred on DreamHost’s end, and now today a separate client of ours has no e-mail. The Wiki.dreamhost.com site was down and the main page was down too. We try to provide our customers with top-notch support and it’s difficult to do so when others don’t provide the same standard of service.

TIL Nobody can be mad at downtime unless they are paying $15,000 a month for service. That Dreamhost doesn’t want the business of anyone who would be more than slightly inconvenienced by downtime. That, according to many posters here, the only people who should host with Dreamhost are little old ladies’ crochet pattern websites and personal blogs about your pets.

That about sum it up O Defenders of Dreamhost?

At the moment both my site and email are working. Email is lagging a bit and I wonder if I’m missing any potentially sent email. I have too many other sites to check to make sure of them all, maybe someone else will know if it is all working with their sites.

@dlocutor +1

Totally agree with Scott.

@Austin, yep exactly.

If you are looking for reliable web and email hosting, message me!

We are back up! THanks guys :-)

I really thing you guys should remove the 100% guarantee from your website

I’m back up. Thanks DreamHost!

I really thing you guys should remove the 100% guarantee from your website!

mrh7184 _at_ gmail

If you are looking for reliable web and email hosting, message me!

mrh7184 _at_ gmail

Scott, you are an idiot, obviously there is always going to be some type of outage, but dreamhost promises this:
The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

We have been promised something, and dreamhost is NOT delivering.
I’m glad you are having a good laugh at everyone’s expense. I know how to host my own servers, and believe me I am seriously contemplating doing it for the majority of my web customers at this point as dreamhost has seriously let me down over the past few months.

Right there with you Scott.

@dlocutor ??? kinda funny but what? what manual? my sites are back online.
@dlocutor and who is holon?
thanks

Not good but not the end of the world. Thanks for the updates. Good luck and I’ll do my bit by not pressing the resend button every 3 seconds.

you guys need to cheer up and read @DEVOPS_BORAT on twitter https://twitter.com/#!/DEVOPS_BORAT

sites – up
ftp – up
mail – up
cpanel – up

see this is why im loyal to you guys :D .. everybody gets in trouble from time to time right…

Thanks DH

Just made the move to Amazon EC2.
After the security breach and increasing downtimes I had no more patience.
Now everything works just fine, ist even much faster and in the end even less expensive.

question: i sent a few test emails to a site to test while offline.. now that we are online, i still haven’t seen those emails come through. just current emails. did i lose those?

My site is back, but panel is still out.

Must be nice for all the SHARED site to be working… my VPS which I pay a premium for is still down.

emails came through, it just take some time

January 30th, 2012 at 9:17 amLong Time User (But...) Says:

it would be helpful to get some explanation of all the downtown and how it’s not going to happen again.

@jen, most emails are resent on retry after xxx minutes then give up after x days. So emails are less likely to be lost. just delayed.

Again?!?!

I’ve been with Dreamhost for years but my clients are asking me to change providers because of downtime/speed. Sorry guys but I need to start considering for alternatives…

Sad Client, click my name, then be instant happy. we all admins are.

@thanks jason. :)

All work now ….

Panel is still down. Not fixed at all.

Sites and panel are down.

those who are running sites with a lot of static content like blogs, might want to check out http://www.cloudflare.com its free for one site and it also shows your visitors any public pages it has cached. You have to research regularly yeah?

Back now, thanks. Pretty decent service for the price I pay, I think.

Summary Statistics Report for Blog front page

Since: February 9, 2007
Outages: 134
Total Uptime: 99.693%
Total Downtime: 0.307%

Year Outages Uptime Downtime Avg. Response Time
2012 4 99.728% 0.272% 0.243
2011 16 99.836% 0.164% 0.227

Year Month Outages Uptime Downtime Avg. Response Time
2012 January 4 99.728% 0.272% 0.243
2011 December 0 100.000% 0.000% 0.264
2011 November 0 100.000% 0.000% 0.186
2011 October 2 99.789% 0.211% 0.492
2011 September 0 100.000% 0.000% 0.181
2011 August 5 99.213% 0.787% 0.180
2011 July 0 100.000% 0.000% 0.203

Because I was told I’m the one in the wrong for getting bothered by downtime, I was still frustrated, so went to my favorite social media and liked and upvoted and commented there instead. Keep doing Dreamhost favors by telling off upset clients at dreamhoststatus, and telling them why they’re idiots for hosting with Dreamhost in the first place. That’ll learn ‘em.

Last week we had 3 days without the ability to send mail. Finally got it fixed – with NO explanation of why or how it got broken in the first place. NOW send mail is not working again. This time I’m getting: Server error: ‘451 4.3.5 Server configuration problem. I cannot get into my Web Panel. My client is really angry and I’m freaking out. I loved DreamHost! – PLEASE don’t make me move all my sites to another hosting firm – get this problem solved asap.

I seem to be back up. Thanks for your quick response! hopefully this is not a recurring issue.

all my shared and VPS are back
I am still moving
just need to finish my backups

http://www.knownhost.com/vps_packages.html
1200MB RAM for $45/mo ($39/mo every mont with 15% coupon)
960MB RAM only $35/mo ( $29/mo every month with 15% coupon)
1472MB RAM $65/mo
1600MB RAM $85/mo
Been with them 4 months and never down.
Cpanel is $5/mo extra but with 15% discount coupn its practically free.

QUICK! as soon as even a little bit of the services come back up, start hammering them all with requests and such, that’ll help the recovery process a lot.

“The world’s an imperfect place. Screws fall out all the time.” – John Bender, “The Breadkfast Club”

those who are using google apps for email, it should be working fine.

What’s happening on 2012? This January looks like a very black month to Dreamhost…

That’s it. We’re taking our business elsewhere.

All back up and running for me. Thanks guys.

Be good to hear some follow-up on why the recent spate of problems and general strategy for future avoidance. As a satisfied customer for 6 years or more, I am (was) in the process of moving some more business critical customer apps over to you … you’ve made me nervous.

“Keith Says:

They can’t keep the site up long enough for me to move it.”

Thank you for that Keith .. a bittersweet laugh.

About your up-time guarantee. If I can’t access your website to report a problem issue (open a support ticket), how is the “timer” supposed to be activated regarding your 100% up-time guarantee? Are you going to proactively refund customers for these outages this past month? In the years since I’ve been a dreamhost customer, this hasn’t happened where we’ve experienced an outage. This January, SEVERAL outages, this is unacceptable. Would appreciate a response.

The DreamHost 100% Uptime Guarantee

DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today!

@millionaire
its still there, you have to click on “satisfaction guaranteed” tab and then it pops up.

January 30th, 2012 at 9:37 amNetrapal Singh Says:

Team,

Whats happening in this month ? what sort of problem are you getting … Hardware? Software ? or Network ???
You must fix it soon .

Thanks All!

Don’t feel the haters’ Sh!t happens. I know..I’ve hosted tons of other places..it happens anytime, anywhere, it just does!

Thank you for resolving quickly.
30 minutes (though it was down longer) after notifying my clients their sites were down..i was happy to send them an update that all was resolved. :)

This is whats going on folks:

NOW DREAMHOST IS FREE FIRST YEAR!!!!!

So now they are becoming as BAD as 000webhost or any free hosting company.

Good Bye DH.

Panel is up for me. Bullet dodged.

Panel up… FTP still down… on VPS.

I’m experiencing errors when I want to access my blog URL and its admin area since days. Today the error reached its major downtime. During the last days I had trouble to access mentioned site for some hours, today no access is granted at all.
I’m currently on the free testing phase, but I wanted to keep my blog running at your service.
Does the problem has to do with the problems you’re suffering from or is it because of the free testing phase?

I feel a big credit coming our way….

Dear Justin and other Dreamhost people,

It’s ofcourse hard to be happy right now; service has been wobbly for quite a while and outages come at a bad time no matter when they come. I do appreciate however that we get very regular updates on this status and that we can see how things evolve on your side instead of having to wait for news. Keep up the good work.
People who host mission critical sites on a hosting company like dreamhost should probably move but for most of us I guess two hours of downtime is not life-threatening. The situation at dreamhost is not bad; I generally get responses to my questions fast enough for me to be on top of things, I’m generally a happy customer but I hope the situation improves fast. I have been a customer with Dreamhost since 2004 and have seen the situation go downhill as of a year or so now. I hope this situation will pass…

I have hosting with another web host for some of my other websites and have not seen the number of issues I have dealt with here at Dreamhost. This was the final straw, I moving the rest of my websites over to http://www.monstermegs.com . The issues have just become too numerous lately. With the security breach last week and several other outages, it’s just time to get out while I still can.

Hurray! Thank you Dreamhost for your frequent updates and hard work in fixing this serious outage. Here’s hoping I won’t have to visit this blog for a while.

> There still will be some cleanup from the network being down, but should be minimal and just a few servers
Looks like I am the lucky one and I am a few!

January 30th, 2012 at 9:49 amChristopher Bright Says:

As for the talk of moving to other webhosts.

When other webhosts get with the time and start offering IPv6 hosting, only then will I even think of moving. At least Dreamhost knows the future of networking.

My VPS wasn’t down till I posted yet another support ticket about not being able to FTP after YESTERDAYS debacle… now its down. Hmmm? Fixing it (should I hold my breath?) Naaah

Been using Dreamhost for a while now for both non-profit organization websites and business sites. I can say that this is not the norm. While yes, I do suffer some business loss while my site is down, I have other backups in place that allow my customers to be able to get to me. If you are concerned about your emails, hell, Dreamhost gives you the option to host emails with Google. Like others have said, you feel that Dreamhost is not working for you, then move. There are plenty of other host out there.

I have some customers that use Magnolia Sweets, http://www.magnoliasweets.us/, which offers various levels of hosting. They have shared hosting starting at $5.83, VPS starting at $14, Semi-Dedicated starting at $50 and Dedicated Servers starting at $135. They swear by them and I have a couple of sites that I use for testing hosted there and they do quite well.

Otherwise, quit dogging Dreamhost.

So my sites and FTP are working but test mails I sent during the outage have not come through my email (IMAP) so I’m concerned that customers who mailed me during this time didn’t get through either (which means lost orders/business/money for us). Is there anything that can be done to retrieve the mails or are they probably lost for good?

fed up. looking at leaving and taking our 4 dreamhost accounts with us.

Dear Colleagues
Downtime is never good … but thanks for getting things back together in reasonable time. I am always relieved when I understand it is not ‘my problem’ but yours, and you are going to fix it! Compared to doing these things ‘in-house’ it is still ‘Dreamhosting’. Thanks.

my mail is back but I am not getting the mails Im expecting..
Do I have to inform people to send them again?

Particularly worrying (ref @Chris) is that even Virtual and Dedicated Private Server customers seem to have suffered.

At the shared service level the stats (ref @Cathryn) still seem very good for the price.

Um, all my shit is still down.

my shared FTP have been down all day, but one hour. Can’t work, too bad

let it back soon plzzzzzzzzzzzzzzzzzzzzzzzz

1.5 hours is “promptly”

Fifth time in January we’ve been down. This is not acceptable. Especially, not on a Monday morning. We want some real answers now, and I agree with the others, we want to know that this is being fixed reliably or you give us no choice but to move.

I can’t connect via panel of dreamhost, could you please take a look into this matter ?

AGAIN DOWN??? SOME cash discount??????

Why is everyone saying its not DH’s fault? This much downtime is NOT acceptable. No matter how little the customer pays.

A hosting company should be able to provide 100% hardware uptime! Why? Because with today’s technology, your web site shouldn’t be linked to one server. It should be spread across, scaled, and avaialble.

If Rackspace can host a VPS site for 10.99/mo and provide 10GB storage, so should Dreamhost. Dreamhost actually charges more than Rackspace for VPS and DH has more downtime!

See where I am getting at? It’s DH’s infrastructure that sucks. They need to fix this. The type of customer shouldn’t matter. The service should!

I will agree you should not have your business hosted on a ‘shared’ account. At minimum VPS or dedicated. But the excuse you are only paying 9.99/mo is bull shit.

8pm O.O! when will the database be back? do we still have to wait for more than 10 hours?

January 30th, 2012 at 10:20 amChristopher Bright Says:

I wonder how many people are actually going to notice the fact that as they just said, it was a DDOS attack. This, of course, means, they can’t do much more beyond fix it as fast as possible.

We were down 9 hours yesterday and now down again for a couple of hours this morning (and still down). Not to mention a good 10-12 hours last time. Do people think this is acceptable for a host? We are new to Dreamhost and have never had problems this severe with other providers.

@Christopher Bright

That’s because their network security sucks. Microsoft, Apple, Dell, etc get hit 10000x harder each day and still maintain uptime. Heck, Wikipedia, a company with only a handful of employees still manages to stay up? Why? Because their admins know what they are doing!

WHEN WILL SOLVE THE PROBLEM

I think dreamhost’s claim to 99% uptime just went to 99% downtime. My clients are ready to jump ship. Not good at all for business.

January 30th, 2012 at 10:27 amChristopher Bright Says:

@Chris 99% downtime would require being down a lot more than it has been.

Just remember a lot of us are on VPS or PS hosting and paying a lot of money, not $10 a month. I keep saying I’m going to leave every time this happens, but it’s just so much trouble. I’m giving them one more chance, it sounds like in their post they are taking this seriously. One more major outage and I’m actually going to switch hosts this time.

I know there are issues, but Dreamhost actually tells you what the problem is and what they’re doing about it. I love that openness.

There’s also a difference when mentioning 99%, 99.9%, etc. in regards to uptime

If it’s up… It’s down… per year
90% 876 hours
95% 438 hours
99% 87 hours, 36 minutes
99.9% 8 hours, 45 minutes, 36 seconds
99.99% 52 minutes, 33.6 seconds
99.999% 5 minutes, 15.36 seconds
99.9999% 31.68 seconds

Please move my website to dreamhoststatus.com servers. They are always up it seems.

my humblest apologies for the network outage i caused.
i’ve reconfigured a few things, deleted a few more admin users, moved the yellow lever to a neutral position, and turned the purple knob to 4.
we should be good to go in a minute.
meanwhile i’ve finished chapter one of the manual. it had a picture i want to share with you:
http://upload.wikimedia.org/wikipedia/commons/0/0a/Tr%C3%A5dtelefon-illustration.png
again my apologies to all of you, i’m sending more refund checks,
they should be in your mailbox .. go check wudya!
wudya is holon’s brother

down, then up, then down again… un-reliable dreamhost.

poor planning or reliance on a $10, $20, $50/month web host for “mission critical OMG NO DOWNTIME” website is downright stupid. Thanks for the hard work, DH team.

Been a Dreamhost customer since 2003 and you guys are going downhill fast. This whole month has been one thing after the other. Very disappointed. Seriously considering looking elsewhere. I’ve already switched all my email to gmail since you can’t seem to keep email going for long. Whatever changed, bring it back. Downhill and fast..

Doesn’t this bring back flashbacks? Dreamhost used to have outage problems a lot. In the past few years (since the double-billing debacle) they had improved greatly, and I had convinced people to host here. I hope this isn’t a trend. I don’t want to have to go back to being ashamed of my webhost. It’s still the best host for my large assortment of domains I play with, but it might be getting to be time to spin off the more successful ones to a more reliable host.

For those of you who said you’ve been here forever and there’s never been a problem before, you have short term memory loss.

Apparently you can’t complain about downtime unless it’s mission critical, and you shouldn’t have a mission critical site on Dreamhost because it’s cheap, therefore you can’t complain about Dreamhost downtime.

That’s some good logic.

One of the biggest sources of Dreamhost new clients is referrals. I’m going to tell them – sorry, IF YOU WOULD BE UPSET BY DOWNTIME, ***DO NOT*** host with Dreamhost. Because if you complain, they, and their surrogates, are only going to tell you that if you were going to get bent out of shape, that, and I’m not kidding, “You never should have hosted with them in the first place then!”

That’s a quality service right there, I tell ya what.

My wiki is now giving “500 – Internal Server Error”
The log says “Premature end of scripting headers.
Anyone the same?

Our personal development sites: http://www.amareway.org & http://www.iswb.org/ are down. Can you please check and stop any external attack? Thanks!

“…tonight to make some network configuration changes that will mitigate against DDOS attacks…”

Is it just me, or if there were some network configuration changes to mitigate against this attack, why weren’t they in place already? Sounds like a crock of discarded fecal matter to me.

I’ve been here since 2004. Started transferring to another hosting service last week.

Indeed there is, Mark. Small changes in a percentage applied to a large number (such as the # of hours in a year, ~8,736 hours) produce large changes in the computed result!

I have one of my sites monitored by an third-party service. I’ve had that monitoring in place since February 10th, 2011. So I’m a little shy of one year of data, but here are the statistics:

Uptime: 99.80%
Total Downtime for the entire period (~353 days): 16h 44m 7s.

76 “down times”, ranging from 59s as the shortest and 89 minutes as the longest.

99.8% uptime. That seems pretty good to me.

I think Linus has the right idea. There are problems, there are ALWAYS going to be problems, outages, things to fix and improve. That’s just the way _life_ is.

What we like about Dreamhost is that they fix the problems and while they’re working on it they inform us of what’s going on. When I see something not working, I check here and learn what it is, or if they’ve not posted here yet, I send them a support request and I get a response back within a little while.

I admit, like everyone else, I feel impatient and anxious when something is down. And it feels like a big deal when everything I planned to work on that day is suddenly inaccessible. But that time passes–generally within the amount of time it takes for a good coffee break. Or tea, if you’re like me and prefer a good cup of tea to coffee. :).

It’s all good. And it’ll all get better.

OOOOOPS! My Dreamhost mail and everything is up again.

I have sites that are still down. You’ve claimed the issue was fixed?????? This is getting ridiculous.

Are there some servers still down?

I’m offically done with dreamhost.com no live support, terrible service. There is always something wrong. Good bye Dream Host. This is nuts.

Yes, there are. I have multiple pages / URLS that are still down and clients are infuriated. They think it’s my fault.

500 Internal Server Error error

is what I’m getting now.

Today nothing digital is 100%. Do you change your home satellite TV system each time heavy rain storms knock out reception. If a car takes down a telephone pole and killed your phone service, do you scream and change phone companies? Do you change cell phone companies every time you have a dropped call or cell service is jammed, or a thousand people are trying to update their Facebook pages in a football stadium and service is running at 24 kbps? What did you do when your fax number or fax machine went down for a few hours or a day? I hate the service being down but face it people, you will never find anyone with 100% service. That is life.

Competition loves to make others look bad. DDOS attacks bring down networks and make hosts look bad. People jump ship to other businesses. Someone else gets paid. Think this doesn’t happen?

Reporting..al my sites and panel are up!

“Competition loves to make others look bad…”

I don’t think DreamHost needs any help looking bad. They seem to have that covered without assistance.

Of course, I suppose some competing hosting service could arrange a DDOS attack on the chance that they might pick up some tiny percentage of folks leaving DreamHost. But if DH can do something, as they claim, to mitigate DDOS attacks, why have they waited so long to do so? If they notice some arcing wires throwing sparks near a can of oily rags, do they wait until a fire spreads to the whole facility before doing something about it?

My site is still down. Do I need to do anything or will you fix?

all sites still down.

all’s working fine on my sites. Iain, you said it well!!

Some of you people REALLY need to get a grip. DDOS means a Distributed Denial of Service attack. In other words, someone from the OUTSIDE is trying to overwhelm a site or a group of sites hosted at Dreamhost with hundreds, if not thousands of requests simultaneously.

You idiots think this is Dreamhost’s “fault” because your cheesy little websites are unavailable. You have “tuner” with his predictable plug for knownhost spamming the page here, and others saying they are going to move their sites somewhere else.

Good. Leave. Don’t just flap your lips and wave your arms, pack up your shit and get the hell out, because you certainly aren’t contributing anything with your whining and chicken little outlook of life.

Yesterday I sat here reading all of the hand-wringing, teeth gnashing and the rending of garments because of a major outage at Dreamhost. Oh, the thousands of dollars each of these people were claiming to lose!

Really? If you are relying on Dreamhost (or any other web host whose average web hosting cost is $10 or less) for your “enterprise level” web hosting needs, then you are complete fools. Your “clients” are even bigger fools for listening to you.

The “uptime” concept here is apparently misunderstood. 100 percent doesn’t mean that servers don’t fail. It just means that during a set period of time, the NETWORK has remained up and running for that percentage of time. There is NO SUCH THING as a network that is truly infallible, and they will ALL fail at one time or another. The secret to the claims of the percentage of uptime is to pay close attention to the PERIOD of time they are calculating.

The computer you are using to read this will eventually fail. Your internet connection will eventually fail. The car you drive will break down one day, regardless of how well you maintain it. Someone is likely to crash into you some day and there is nothing you will be able to do to stop it. Shit happens.

My site is still DOWN!

DJ Allyn, failing to prevent DDOS is Dreamhost’s fault.

@ DJ Allyn, Which service would you recommend for a medium size business that supports unlimited storage and bandwidth as well as a support for the latest Word Press? Just curious since you were quick to criticize without giving any constructive feedback.

@DJ Allyn +5 thank you
geez i thought i broke everything by “sausage fingerin'” the wrong command ..

When the hosting provider says the problem is fixed and sites are still unavailable even after re-boots on virtual private servers then the problem has not been fixed.

I AM STILL DOWN why do you post messages saying everything is fixed when it clearly isn’t.

I’m assuming the problem I’m having is related to this (not able to log into ftp), even though it’s been hours. Any further updates that will confirm or deny this?

Thanks!

Thank you for your quick response to this morning’s problem. My sites are all reporting 100% service.

phew. so this is why i can’t log into ftp; thought I had drunkenly changed my password again.

VPS still down. I have NO IDEA why you’re saying everything’s fixed. It went down again with this issue, and it hasn’t come back up yet. Good lord.

i’m the dj, he’s the rapper.

“Good. Leave. Don’t just flap your lips and wave your arms, pack up your shit and get the hell out…”

We’re trying, really. It does seem as though DreamHost is trying to drag out domain transfers away from DH as long as possible. I thought only GoDaddy stooped quite that low.

i’m so f’d

Restore FTP/SSH and I will Tebow on my Laptop!

Pleeeeeeeeeeeeeease!

I use my sites to solicit partners for EROTIC ANAL BUKKAKE.

Every minute I’m down, I’m losing out on potential revenue from our new movie, in which I take 500 men and penetrate 500 women and 250 young boys and shemales…

I’m going to move my sites to HostGator, where some stoned Houston hipster can accidentally delete all my stuff and tell me it was a virus on my laptop, as soon as possible! I hear many of the HostGator employees are bi, tranny, or emo-gay as well.

I smell opportunity… or maybe that’s just lubricant…

Mode migrate to other Hosting place plan —> ON

Unfortunately, I’m unable to access the account via my login or the master login (a different person). This is extremely problematic.

Wow, you guys are jerks. Ease up.

Did you guys read the fine print? These comments don’t go to DreamHost. You need to send your messages to DreamHost. They need to know that a lot of their clients (like me) have had their business disrupted.

Chas: They’re not responding to support tickets either. People are trying every method possible, no matter how improbable…

@Tim, @Dan: Same problem here. Hello? Anyone? Please respond to support tickets?

They’re not responding to support tickets either. People are trying every method possible, no matter how improbable…

We sent 10,000 support requests in the last five minutes, HOW COME THEY HAVEN’T REPLIED so mad I’m moving to HostGator!

hey, provide some instructions on how to change the password.
and make a new post rather than just answering on this thread.

I am not going to go through these threads to find the instructions.

@Greg K

Which service would you recommend for a medium size business that supports unlimited storage and bandwidth as well as a support for the latest Word Press? Just curious since you were quick to criticize without giving any constructive feedback.

I would probably go with Spry.com. They are geared more towards enterprise level hosting. They have a handful of “shared” hosting accounts, but their focus is towards the VPS/Dedicated markets.

And that is the key here. Dreamhost is geared towards the shared web hosting, and while they do have a fairly stable VPS and Dedicated lineup, their bread and butter is still shared. Consider them a “beginner” to “intermediate” hosting service, if you will.

If you are starting out with an ecommerce site, DreamHost will be good for you. But if you are relying on your website to be up and running because it is critical to your business — and you cannot afford to be down — you need to go somewhere else.

That said, MOST of the time you will be alright at DreamHost. I have gone for months without so much as a momentary interruption and I have one site with 45k hits and another on the same server with 32k. The past two days have been an anomaly in my service.

Granted, my sites are not critical, and the most I could lose is the number of hits in a day. I also manage “clients” websites and forums on other DH servers with similar results. When their sites go down, they are inconvenienced, and a few are initially frustrated. But depending on how you explain the situation to them, they are quickly calmed down.

I do manage some e-commerce sites, but not at DreamHost. A couple of clients lease dedicated servers at Rackspace and I have four accounts at Spry.

I think that the point here is that with all of the whining about sites being down, and the need to point fingers and blame the host for a situation that is out of their hands is silly. It isn’t as if DH WANTS the failures. They have a product to sell, and it does them no good to be down all the time or oversold on servers.

What a lot of people here don’t really understand is that a shared account is going to be severely throttled. That means a WordPress site with decent traffic will most likely hit the ceiling fairly quickly. Especially if these people are suckers for all of the bells and whistles in the form of plugins on their site. So they told that if they want to get away from those constant 404 or 500 error messages or if they want to speed up their page loads, they first have to find where their bloat is and deal with it. If they are running a caching plugin, make sure it is properly configured. If that still doesn’t do it, then they need to start thinking about moving up to a VPS.

Now a VPS running a WP or any other type of forum software is probably going to take about 500mb of resources. That is why the ceiling was bumped in the shared account. You can shave a bit of this off by choosing Nginx instead of Apache and running a good caching plugin. But understand that the graph provided with your VPS is not real time, but a snapshot AVERAGE.

Another thing: Just because HM posts an update that indicates that they believe they have “fixed” an issue, and your own site is still down, the smart thing to do is go put in a SUPPORT ticket, because they aren’t reading anything here, and they may not know there are still some isolated residual issues still occurring.

A closed mouth doesn’t get fed. If you are still having issues, THIS isn’t the place to report them — your support tickets are.

As I just said, support tickets haven’t been answered since the original 18-hour blackout.

Con una chingada!!!

@Ridiculous

Chas: They’re not responding to support tickets either. People are trying every method possible, no matter how improbable…

PIG — Problems of Instant Gratification.

Which would you rather have them doing, answering thousands of support tickets, or working on the problem and then getting back to you later?

Yesterday, with the big outage, part of one of my servers came back but I was missing a domain. I put a support ticket in and waited. Later, the domain came back, but I didn’t get a reply on that ticket until this morning.

Imagine this: you are but one person in a giant group of people with roughly the same issue. You have registered your issue with the support crew, and they will get to your issue as soon as they can. Maybe, in the course of fixing the overall issue, they will inadvertently fix your issue too. All the better.

But you did what you could do — the only thing you could do — you registered your issue. Now all you can do is wait.

Some issues are quickly resolved. Others take a lot longer to resolve. There are certain steps that have to be done in order to get issues resolved. VPS and Dedicated machines are a lot more labor intensive to get up and running than the shared machines. That is why they chose to get the shared units up and running first yesterday. There are more shared customers at DH than there are VPS and DS customers. The fact that a VPS or DS customer pays more is a silly argument for vying for priority. You are paying extra for resources not an exclusive club membership with extra perks.

I am only pointing out the realities here. A lot of you might not like the cold hard facts, but someone has to bring you down to ground level.

If someone wants to leave, then leave. The smart person would have a regular backup locally stored so that all they need to do is hook up another server somewhere else and upload. Backups are probably THE most important thing to do when you are dealing with computers. Always have current backups. That way, if your anger gets the best of you and you choose to fly off the handle and leave, you don’t have to hang out waiting.

January 30th, 2012 at 5:12 pmStupid for being here Says:

Support tickets are not being responded to for hours.

It’s time to leave.

This problem isn’t about plugins on WP site, it bad service, and I’m not referring to the hardware issue. The Tech support seems to be taking a holiday from responding to tickets.

Apologies are like the offered yesterday show how week of a company DH really is, if they were truly sorry, then they would keep the lines of communication open. Not responding to tickets for hours after the issues yesterday, is just stupid.

Yep, Stupid to continue to treat your customers badly. How stupid, lets see how long they jerk around and keep us in the dark. That is certainly a true measure of their lack of concern for their stupid customers.

VPS is up
Shared down again!!
WTF??

Shared is down
What you say DJ Allyn?
I guess its my fault third day in a row hosting is down?

Thsi oner is for you DJ Allyn:

http://www.knownhost.com/vps_packages.html
1200MB RAM for $45/mo ($39/mo every mont with 15% coupon)
960MB RAM only $35/mo ( $29/mo every month with 15% coupon)
1472MB RAM $65/mo
1600MB RAM $85/mo
Been with them 4 months and never down.
Cpanel is $5/mo extra but with 15% discount coupn its practically free.

CANNOT SUBMIT A TICKET! After I click “Contact Support” and then click the type and click “Next” all I get is a blank page. And chat support is down so I can’t notify anyone. HELP!!!

@DJ Allyn:

Clearly you’re paid by Dreamhost to say this because don’t you see, people are angry! Angry! They feel betrayed and crucified like Jesus, on the cross of unstable Debian installs.

Suggesting that they go pay for specialized hosting is not acceptable. You must recognize their feelings. They want you to pay attention to their special needs.

And instead they got me, and I’m trying to convert them to genderless bukkake. They suffer (but why).

Site is down at 7:00 PST. Not connecting to databases. Is this still a known problem?

The Dream is dead…

good riddance!

@Stupid for being here

The Tech support seems to be taking a holiday from responding to tickets.

Again, which would you rather them do, sit there and respond to your tickets or actually go out and fix the problem.

if they were truly sorry, then they would keep the lines of communication open. Not responding to tickets for hours after the issues yesterday, is just stupid.

Hence the purpose of this status page. They provide updates whenever they can, they try to let you know what, if any progress they are making. Most web hosts won’t even do that much. They will provide you with a Nagios site that shows the status of all of the servers in their network and that’s it. If they do even that.

I have dealt with a LOT of web hosts since 1994, from dial up concerns to large ones like Rack Space. I have dealt with webhosts that have literally shut their stuff down and gone out of business overnight. I have seen it all.

If you want to get some kind of appreciation as to what these technicians do in a datacenter, you should go visit one. This isn’t like a small room with a rack of computers in it, where the tech support guy manning the desk simply wheels his chair over and slaps the box on its side to get it to go again. We are talking about multi-story buildings that take up several acres of hardware and an army of technicians all trying to keep it running. It isn’t like you sitting at your desk and pushing the power button on and watching your computer boot up. These machines have to be started up in phases and each server may take five or ten minutes to completely boot up on a hard boot. Multiply that by a thousand or so machines, and you can see the complexity of it all. Also, because of the power consumption in a datacenter, you cannot just turn on everything all at once, you have to cycle them on a group at a time.

@tuner

Then why are you still here, if you have been at knownhost for four months? Get on with your bad self. Don’t look back. Besides, how soon do you eat up all of that bandwidth? I would eat up that VS5 bandwidth within a week on my site, and the disk space would be gone before I loaded half of my site.

Oh and their uptime?

Year Outages Uptime

2011 41 98.845%

Pfffftthhhh!

I am seeing all kinds of code on the main page after I log in. What gives?

At first I lost my email sending server, then I had to change my ftp password cause it didn’t work anymore and now my whole website is down…. what’s going on?

@DJ Allyn:

Most of these people couldn’t handle the sheer amount of frustration inherent in dial-up and webhosting prior to 1999 or so.

In fact, I think very few of them could handle a genderless relationship.

The other thing you need to mention about datacenters: they sound like airports.

@Jocelyn

I am experiencing the same thing. Have not figured out what the cause is.

Hmm…I can’t open a ticket or get to my site. Hope stuff is fixed soon.

January 30th, 2012 at 8:56 pma client who cannot create a ticket Says:

My PS is up, all services are up, but somehow apache went down 4 hours ago. I am monitoring the system 18-20 hours a day and still get a 4 hour downtime without any clear reason.

I cannot “chat” with support, and I cannot create a ticket using their system. This is getting RIDICULOUS.

How the fuck are you gonna say this problem is resolved??? All of my sites are STIILL DOWNNNN!!!!!!!

DJ Allyn
I dont keep my eggs in one basket I still have 2 accounts with DH,4 more in Europe and 2 with KH so dont you worry about me being here.
KH has worked for me fine and I recommend them.
Besides cheap VPS, they have customer service answering my tickets sent 10PM in under 30 minutes flat which never happened to DH, since 2007 at least.

DH customer service is so sensitive, like a little girls, they will not answer ticket if they feel hurt by your harsh response, so unprofessional if you ask me.

———-
Then why are you still here, if you have been at knownhost for four months? Get on with your bad self. Don’t look back. Besides, how soon do you eat up all of that bandwidth? I would eat up that VS5 bandwidth within a week on my site, and the disk space would be gone before I loaded half of my site.

Oh and their uptime?

Year Outages Uptime

2011 41 98.845%

Pfffftthhhh!

Jim Says:

How the fuck are you gonna say this problem is resolved??? All of my sites are STIILL DOWNNNN!!!!!!!

I am betting that the problem is with you, and your inability to read and follow directions. Case in point:

Look right below where is says, “Leave a Reply”. What does it say? It says,

Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

I’m guessing that they might have some more important things they are doing instead of watching comments blow up on this support blog.

Have you even bothered to read the updates that DreamHost posts above?

a client who cannot create a ticket

My PS is up, all services are up, but somehow apache went down 4 hours ago. I am monitoring the system 18-20 hours a day and still get a 4 hour downtime without any clear reason.

If you read the previous entry about the major outage, the last update says that if you installed Debian’s Apache, you will have to reinstall it.

Above all, contact support.

If, for some reason you cannot submit a ticket, (which I suspect is a browser problem) you can always submit a ticket the old fashioned way — email it. (support@dreamhost.com)

People have to understand that if there is a major issue going on, there aren’t going to be any quick fixes and rapid and personal response to support tickets. “All hands on deck” would be more like it. Putting a support ticket in during the MIDDLE of a major outage like this is ludicrous. They already know that there is a problem, and it isn’t confined to your little box in the Matrix. IF, after they’ve reported here that the issue has been resolved, and you are still down, THEN put a support ticket in. Otherwise, your ticket is just going to be one of thousands, and it will take that much longer for them to get to yours. It is only common sense.

I realize that most of you want to be “kept in the loop” at the very least, and that is the purpose of this status page. Someone will periodically toss a generalized update here whenever they can. Sometimes they will post the fact that the original problem has been resolved and yet a few of you will still be down. Instead of spinning yourselves up like the Tansmanian Devil, put a support ticket in. But like I said just above, the natural inclination whenever there is an outage like this, is to slam the support ticket system with your personal issue and it creates a backlog of tickets of a thousand or more. Yours will take that much longer to get to.

@emosexual

Your idiotic comments are not helpful. I could care less about your proclivities, and I am sure no one else here cares either.

@tuner

Besides cheap VPS, they have customer service answering my tickets sent 10PM in under 30 minutes flat which never happened to DH, since 2007 at least.

Strange, I normally have MY support tickets answered fairly quickly on most occasions. I use the chat feature a lot, and that is even quicker. However, as I just noted in a comment above, when you have a MAJOR OUTAGE as we’ve all just gone through, all bets are off.

What I found interesting was the numerous re-postings you made plugging knownhost yesterday. What, you didn’t think that one or two of them were enough?

I have dealt with dozens of web hosts since the early 1990s. I have seen some good ones and have suffered through some absolute bad ones. You usually get what you are paying for. Dreamhost is one of the better ones in its class and price range. But they are mostly meant for the casual and beginner-to-intermediate market, and not considered to be an enterprise-level class.

DreamHost does not lose its network often, but it has. And it will again. So will every other web hosting company. That “uptime” percentage you see web hosts advertise is NOT based on an individual server, but on their entire network usually by the YEAR. Individual servers go down all of the time. I don’t care what hosting company you are with. Routers go down, connections are lost. But the NETWORK will usually stay up and running, and that is what all of these companies base their uptime on. 98 percent over a year isn’t all that great. That means that in a year’s time a network was down for a cumulative period of about a week.

DreamHost doesn’t have that bad of an overall uptime rating. Since 2005 they’ve had a overall rating of 99.632% Here is a helpful listing of webhosts and their overall rating:

http://uptime.besthostratings.com/webhosts-uptime.php

Here it is again sorted by the best on down:

http://uptime.besthostratings.com/webhosts-uptime.php?order=uptime&page=1&sort=desc

Understand that this is only talking about the NETWORK based on one or a handful of monitored sites. Just because a host has a perfect or near-perfect overall uptime does not mean that they are great hosts, or that their services are geared towards YOUR needs.

Most people at DH run WordPress sites. DreamHost is optimized to run WordPress specifically. Other webhosts are geared more towards other applications, and WordPress may or may not run well on those configurations. (and I am using WP as an example)

The point is, just tossing out web host alternatives based on uptime is silly. Pricing and configurations are important also. Datapipe, for example is at the top of the list on uptime according to that list above. 100 percent since 2005. It also costs about $800 a month for its smallest service that consists of 500mb RAM, 10gb storage and 500gb of bandwidth. And that is “unmanaged”. But they are a cloud service which will allow them the ability to be “up” all of the time.

This is NOT true: “If, for some reason you cannot submit a ticket, (which I suspect is a browser problem) you can always submit a ticket the old fashioned way — email it. (support@dreamhost.com)” It will just be returned to you with this in the subject line: Your support message was NOT received.

When there’s a downtime, this place is the most happening message board on the net. Awesome!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Easy/Simple One Click Installs temporarily offline ( RESOLVED )
1:32 am

Easy/Simple One Click Installs temporarily offline ( RESOLVED )

Posted (January 30th, 2012 at 1:32 am PST) by chalise

Our admin team has had to temporarily take our Easy/Simple One Click Install services offline.

To see if you are using an Easy/Simple One Click Install, go to Goodies / One Click Installs in your panel and click “Manage Installed Applications”. Any installation that says “Simple One Click App” to the right of it will be affected. This will not affect any other One Click Install services we offer.

We will update this post again when our admin team is able to bring this service back online. Thanks for your patience!

Update Jan 30th, 4:03pm PST:Our admins have informed us that the Easy/Simple One Click Installs are back online. Sorry for any inconvenience this has caused. Your patience is greatly appreciated. If you continue to encounter problems with your one-click software, please contact Tech Support with the details. -Brian S

This entry was posted on Monday, January 30th, 2012 at 1:32 am and is filed under General Outages, One-Click Installer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

88 Responses to “Easy/Simple One Click Installs temporarily offline ( RESOLVED )”

It has been 24 hours since my website went down! What can you do to bring back on line? I have a MAGAZINE and I publish every month I’d probably lost “rank” and for sure money because of this! Please advise…

Webmail: ok
Ssh: no
Wiki: no
Web: no

Anyone?

ALL of my websites are having trouble loading now.. what’s up?!

Any idea how long “temporarily” is going to be?

Thanks,
Tom.

Is there a general outage? Emcquest.com is down and I’m guessing this is more than just what you wrote about. Please advise.

What you going to do about the outage? 28 hrs and counting!!! Maybe tweeter is the only way to vent and prevent other people from making the same mistake I did!!!

I’ve got a one-“click” install for you. Just take off your pants and/or skirt.

Gender is such a limiting concept.

ded in the water over here .. webpanel broke ..
me thinks the captain jumped ship

trying to reach panel.dreamhost.com and all i get is:

The connection was reset

The connection to the server was reset while the page was loading.

panel down, webmail up and down. please dreamhost, get it together.

How about the entire panel is offline???

panel.dreamhost.com is down. I need this to be fixed!!

As stated above – Panel is down in addition to One-Click apps being down. It’s disconcerting to not see an outage notice for Panel. What else is not being mentioned?

Ey the webpanel is DOWN!!!

FTP down now too…

everything is down. fix now!

everything is broke
i wanna fix it but don’t know how

Someone needs to come up with a “leave dreamhost day” I’m looking to move all my sites off of dreamhost today. That is… once the panel comes back online………

Down and out. Are you sure it’s jsut that?

Hopefully with enough comments this issue will climb the priority list.
Web: down
SSH: down
cPanel: down

I can’t connect to any sites, nor dreamhost.com :/

MY SITES ARE DOWN, SO THE PANEL! I CAN NOT CONTACT SUPPORT!

SEEMS LIKE JANUARY HAS BEEN A TERRIBLE MONTH FOR DH

January 30th, 2012 at 8:07 amMark Cutsforth Says:

Everything is down, even http://www.dreamhost.com

FTP: no
Mail: no
web: no

happy customer: no

Even DreamHost’s main page is down right now. One would think that’d be enough to get them on top of it a little quicker than normal, eh? =)

I’m getting really sick of this. I’ve had CONSTANT issues for about a month. They were trying to “help” me but they haven’t answered since Thursday night (Today is Monday) WTF!?!?!?!?

same with me, no panel, no email…

Meltdown on my end too.

Too busy to deal with a complete outage!!

Note to self: move email to gmail.

All of my sites are down and I cannot connect to the panel. :-(

Everything is down. Nothing is posted in the status.

Will be changing hosts in the near future.

Mail is down (POP/imap/webmail)
SFTP is down
Panel is down
Site is down

In other words everything is down on multiple accounts!
Some DO NOT EVEN HAVE 1 click apps, as they are just for email!

All of my sites are down as well, can’t connect to panel.

does anyone have experience with moving hosts? is it a problematic situation? i have so many files stored with dreamhost that i am hesitant to move but i just can’t deal with all these problems

No IMAP mail
No webmail
No webpanel to put in a support ticket

Good morning, DreamHost! And a Monday, too.

Ginger
Off to think positive thoughts and refresh pages

I’m down, too — no panel, webmail, FTP, or website.

Everything is down. No webpanel. No ssh. Sites hosted are down.

Everything is down. What’s up….

All of my sites are down as well, can’t connect to panel!
DH is horrible this last months…

Anyone here recommends a new host?

this page is working .. be patient, i’m reading the manual now

Panel dreamhost: no

Webmail: No

Sities: Yes

Fixing: Idont know

Please Helpppppp

This has been a constant problem. What is going on. Do I need to change hosts? Also, they have been terrible about answering my emails. They were great for the first few responses but haven’t answered in 4 days, wtf :(

January has been the worst month that I can remember. Sucks that I prepaid for a whole year.

yep, everything is *down* now.

not even the web panel to report the outage…

Seems like you are having trouble keeping your servers online frequently now, what’s going on? Do I need to look for new hosting? Not happy about this…

panel.dreamhost.com is down
dreamhost.com is down
DNS time out

Cannot connect to anything…Dreamhost used to be a good service. What Happened?

Obviously there is an outtage of some sorts people. All of my sites are currently down, too.
This should get resolved soon, Dreamhost is really good at it. If they were spending time responding on here I’d be like …”what are you doing on here!” LOL

Dreamhost rocks you just dont know till you have try other places…and I’ve tried all the major ones. Accidents Happen. Welcome to the real world! ;)

What a wonderful day to start a week, and of course as always Dreamhost isn’t reporting on the fact that they’re completely down. All sites, mail, webmail, panel, everything are inaccessible. Again. And, without the panel, there’s no way to post a ticket to try and get a response — and they can’t be bothered to actually post any kind of updates here about it. That would make too much sense.

Everything is down AGAIN. First the VPS out of memory errors that plagued you all fall long, now two days in a row of no sites. I can’t even get into Panel. Fix it or offer free phone support so we can call you and bug you about it.

All of my sites are down as well and can’t connect to the panel. I have several sites offline right now!!! Please give us an update on when this will be fixed!

dhstatus DreamHost Status
Major Network Outage

twitter blast .. on a page four

sunspots

All sites down

Can you please provide another way to file support tickets other than http://panel.dreamhost.com? Because when the panel is down, I’m left with hijacking a comment thread of an unrelated issue on this page!

Otherwise, same as everyone else. All of my sites are down, the panel, SSH, etc.

come on man, this is taking too long

Another complete outage, everything down once more. If it were not so much work (websites, mail, DBs), I would have moved already. Dreamhost offers a lot of great features and storage space, but stability has been poor in the recent months!

Same here. YOU GUYS SUCK!!

For everyone else — usually, you can go to http://webapphelp.com/ (which is owned and operated by Dreamhost) for help… Unfortunately, that’s down too! :(

All my sites are down including mail! Dreamhost 4tw!

And, just to add fuel to the fire… Who hosts this site? I think I should switch to them. This site is NEVER down!!

All my sites are down and I cannot ssh into Gabarone nor report the problem officially through the Web Panel because that too is down. It’s 11:23am Eastern (8:23 Pacific) and I am not sure how to report the outage…

No websites (labelle.org, scottnolan.org, vampyr.org, etc).
No email (neither webmail nor imap)
No ssh access
No https://panels.dreamhost.com

LOL @ “And, just to add fuel to the fire… Who hosts this site? I think I should switch to them. This site is NEVER down!!” =D

All of our hosted services are unavailable also! What’s up? This is the third time in less than a month. We’ve never had so many issues with Dreamhost before. Please correct this issue, and make it right.

What happened to the status that was here on this site 15 minutes ago saying there was a total outage? Did they remove it thinking nobody would notice?!

Ditto…was in the process of cleaning up sites from hacks and lost shell, sftp, etc. All sites now down.

This is just too much … Websites down, mail dead, what’s going on??

All my sites are down! I can’t receive email. Please help!

Best of luck. Hoping you will fix this soon.

I confirm, all sites, include dreamhost are down, and email is not working

Guys, you have all of our sites down! this is not some one click thing. we have 6 sites down and just sent out emails! Do you guys not have a way of using another server to host sites while you do your repairs? This is becoming more frequent which is not good for your business or ours.

maybe it’s anonymous……

moving host had enough of this…

All sites down for the second time in 12 hours. Hope something is being done to rectify the situation.

We need things to be working and there not!! I agree Aran we will be moving A.S.A.P!! Gotta wait to get my files though!!

A status update saying that the situation is acknowledged and you’re working on it would be nice.

Is there a one click install for DH2?

Everything down. Again.

Only this time I can’t even contact support.

all sites down – (just supporting the issue into TOP PRIORITY.) If I’d a known, I’d a stayed in bed. (hiding from customers)

Q: What’s a webmaster without his sites?

while we’re here…. anyone’s been looking at hosting on AWS?

Been a customer for 5 years and shocked to see where the overall quality of service has fallen to.

The costs of hosted plans with DH has not gone down, but the quality of service has.
Who cares if your storage and bandwidth usage goes up when core services fail for over 48 hours? Clients are furious.

The DH apology was the first step in accepting responsibility.
I recommend taking some other actions to show your customers you care enough about them to do something.

Sorry about the downtime you experienced this morning. Please see details posted at http://www.dreamhoststatus.com/2012/01/30/connectivity-issues-in-one-datacenter/
If you have questions about the one-click installs being offline, please contact support through the DreamHost panel.

The other issue is that backups are limited to 4GB, so the entire platform is somewhat scary.

Sodomy: Yes
Penetration: Please
Webmail: ???
Coprophagia: Ohboy
Bukkake: Yes
SQL: who cares

Are the One Click Installs offline again? I also can’t submit a support ticket in the web panel.

Hey there,
I’m still having problems with my one-click installations. I just bought another domain, and was interested in installing WordPress, however, it does not work. Nothing but code shows up on the page. When I try to submit a support ticket I am greeted with a blank page. The only way I know how to contact Dream Host is through this forum.

Please advise.

Thanks!

One click still broken for me…

I’m with Kae Lani Kennedy.

one click is still broken for me as well.

One again “alouettes” Web server is down, please take a look and let me know. Why is happening with more frequency that usua?. It have been couple times in the last 3 weeks.

Regards,
Joseph

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Webmail login issues (RESOLVED)
12:14 pm

Webmail login issues (RESOLVED)

Posted (January 29th, 2012 at 12:14 pm PST) by danwy

We’re currently experiencing some problems with one of our databases that services our webmail machines. As a result, you may be experiencing intermittent login issues when trying to login via webmail. We initially thought that this issue may have been related to another posted outage, but upon discovering this was its own issue we decided to make its own status post. Our admins are investigating the cause and will work on resolving this ASAP. We’ll update this post as soon as we have any further information regarding it.

Update Jan 29th, 1:40pm PST: These issues are resolved now. We needed to restart dovecot on our webmail machines and have not been seeing any login errors since that was done. We’re not quite sure what caused this to prevent the database from connecting, but we’ll be investigating that further. We’ll be resolving this post now, as we have not been able to duplicate any errors since restarting the dovecot service. If you continue to have problems with ‘unknown username or password’ errors, please contact support to have them investigate it further for you. -dan w

This entry was posted on Sunday, January 29th, 2012 at 12:14 pm and is filed under General Outages, Webmail Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

26 Responses to “Webmail login issues (RESOLVED)”

Yep, I’m affected by this. Can’t log-in to any email on any site I have.

I can login in to webmail from home, but not via work connection. Same account, same password, two different results in two different places at the same time, strange behaviour =)

SEVEN HOURS to resolve a login issue?!

This issue has been going on since about 2 am mountain standard time, I was up doing some work on customers web sites and have only been able to log into mail a couple of times, it is now 1:36 mountain, so it has been going on for almost 12 hours now, any updates would really be appreciated.

ahhhh YES!! so this is why i couldn’t log into my mail, i changed my pass also. When I did i got in, but then after a few mails, it made me log in again, tried with new pass and i couldn’t get in. FIX ASAP!

here is a surprise. wondering if you are going out of business or something

A lot of downtime lately.

Yes I can’t believe the amount of problems with this. I’ve been having webmail issues for weeks. Along with the security breach and change of ftp passcodes that has taken hours and I still am not finished. Then they wanted me to figure out the webmail issue….come on really? I’m getting seriously sick of paying for this downtime.

Welcome to DH’s webmail. After years of days without mail, we changed to using Mozilla Thunderbird as an IMAP client. Nice. Still need the Squirrel to get to your Junk folder. DH has been a fine host, but they just don’t put much effort into webmail.

when can i access my email???

I have also experienced stability issues with webmail more frequently lately, unfortunately.

p.s. i was able to log in briefly a few hours ago, but cannot now

Hugely disappointed with dream host today

You have been decent hosting for the past few years, but in recent months your service has been unsatisfactory. Repeated problems with webmail is driving us to look for another hosting service. Owners of the company should be looking for a better management team by now. Your monthly emailing to customers needs to explain these problems, what was done, right and wrong, and how you are fixing them instead of continuing the less-than-humorous drivel.

Was affecting me. Works now. Always worked via client, just not through webmail.

Unfortunately, this weekend, I’m in the middle of a promo and my website is down. Anyone have any idea as to when the server will be up? Paul foodonpurpose.com

This is so disappointing. Twelve hours of downtime?!

hear they are going out of business

I use the MX google email service. Does it influence

up again.wooohoo

webmail is down again!! FUCK

January 30th, 2012 at 8:41 amRichard Howland-Bolton Says:

Down for me too!!!!

My webmail is down AND I cannot get into dreamhost.com. Please fix!

Getting intermittent login errors while trying to open the Sent messages folder in squirrelmail. And now the panel is down.

I can login to webmail (via roundcube) but get kicked back out to the login screen after a couple seconds…

Can’t comment, am busy moving my 400 sites that make gazillions of dollars over to AOL “home” hosting, where my $8/month goes a lot farther!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down (RESOLVED)
3:35 am

Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down (RESOLVED)

Posted (January 29th, 2012 at 3:35 am PST) by jj

The apache (web), SSH, and ftp, and services on a subset of our VPS and dedicated servers are currently down. FTP services on some shared servers are also experiencing downtime. Our system administration and data center operation teams are currently on the case and we are attempting to restore services as soon as possible. Further updates will be provided.

Please note that this only affects web VPS/dedicated and shared web server FTP services, no other services or servers (MySQL or mail, for example) are affected.

Update Feb 1st, 4:30pm PST: We are no longer seeing any fall outs from Sundays’downtime, so we consider this incident resolved. Thank you again for your patience and understanding while we were working on this. If you have any questions about this case or need further assistance, please contact support, so we can assist you. – Andrea

Update Jan 31st, 11:24am PST: We’re still seeing a handful of VPS servers in need of help. If you do *not* have an admin user on your VPS, a simple reset from the DreamHost panel should get you going again.
If you do have an admin user on your VPS, or the reset didn’t help, please contact support with the details, so we can assist you. – Andrea

Update Jan 30th, 12:03pm PST:
Just a quick follow up to address some of the problems that appear to be still around: If you’re on a VPS and installed Debian’s mysql-server package on your web VPS, your data is still there, but you will need to re-install that package. If you notice additional downtime, check if your mysql libraries and “mysql” and “mysqldump” utilities are uninstalled from your web VPS, and please contact support, so we can assists you.
If you installed Debian’s apache instead of ours, to manage your own apache, you will have to reinstall it. – Andrea

Update Jan 29th, 9:40pm PST:

From Simon Anderson, CEO, DreamHost: My sincere apologies for the downtime experienced today by many of our dedicated and VPS customers, plus some shared customers. I know that this has been a poor customer experience for you. Almost all services are back up after an intense effort from the DreamHost dev, admin, data center and support teams. I was involved in the coordination of our efforts today and now am able to share what happened, and what we’re going to do to reduce the risk that it happens again.

We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers. Our monitoring and support team flagged the issue fast, and we scrambled our admin, dev and NOC teams to reinstall the packages that had been removed by autoupdate, reboot servers, fix package dependencies, and test that individual services were live. Given the number of services affected, this took a long time to complete. Rest assured we had all hands working on the issue, but I know it was still a frustrating experience for customers.

To mitigate the risk of anything like this happening again, we’re immediately switching off autoupdates, and moving to a manual process where we’ll only push out Debian updates after significant testing. There’s always a balance to be struck between speed, efficiency, security and issue prevention, but this event has shown us that we need to take a different approach. Again, my apologies for the downtime experienced today. We’re acutely focused on adjusting our processes and systems to ensure we do a better job going forward. – Simon

Update Jan 30th, 5:45am PST: We are continuing to address the individual VPS guests that are still experiencing issues. Again, we have found that an Apache restart is resolving most problems; this can be done manually, or contact support and we can do it for you. If you have any other issues, please contact support and we will be happy to help. Thank you, once again, for your patience as we work toward a full resolution. – Chalise

Update Jan 30th, 12:08am PST: There are still a few VPS guests we are still continuing to work on. An Apache restart is fixing most issues, which you can do manually, or contact support so we can do it for you. If you have any other issues, please be sure to contact support for assistance. – Chalise

Update Jan 29th, 7:57pm PST: We are almost done with fixing up the affected servers! A handful of VPS servers are still being worked on at this time. If you can access your VPS, you can restart your Apache instance to speed up this process (you can contact support for help with that, or to have us do it for you). All of the Dedicated Servers are online now and working! Please contact Support for additional assistance! -Patrick

Update Jan 29th, 6:32pm PST: We’re continuing to fix up one by one the remainder of the machines that are still affected by this outage. We appreciate your patience while we’re working towards the resolution. We’ll keep updating this post with more information as they come in. Very sorry about the inconvenience this is causing. – Andrea

Update Jan 29th, 5:15pm PST: Not much to update since our last post. Our admins are still working away on the remaining VPS and dedicated servers that are having issues. Steady progress is being made, but we still don’t have an estimate on when things will be completely resolved. We will update this post shortly, or as soon as we have any more specific information. -dan w

Update Jan 29th, 3:45pm PST: Our admin team continues to go through the remaining VPS and dedicated servers that have broken services. For those asking why we focused on getting the shared servers up first, there wasn’t a conscious effort to do so. It just happened to require less manual intervention by our admins, so things were resolved much quicker. We’ve had a bulk of our admins working on restoring dedicated and VPS servers for the last few hours, but there are far more of those affected and they take longer to resolve because they require more manual intervention. Our admins will continue to work on this until every server has its service restored and is functioning correctly. We’ll update this post shortly with more info. -dan w

Update Jan 29th, 3:45pm PST: Not very much to update from the last post. There’s still a handful of dedicated servers that we’re still fixing MySQl services on. There’s only a few shared hosting servers left that need to have FTP fixed on them. VPS servers continue to receive the most of our admin teams attention and we’re getting those configured and restarted as quickly as we can. We’ll update this post again shortly with more info. -dan w

Update Jan 29th, 3:00pm PST: Our admins believe they have the all of the issues with MySQL on dedicated servers now resolved. We’re running some tests and will be cross referencing it against a list of affected servers to see if there are any still not functioning. We’ll be able to fix up the remaining ones manually then. We have less than 20 shared hosting servers that have FTP issues still and our admins are getting those fixed up at a steady rate. VPS servers are being fixed up at a steady rate as well. We don’t have an update from admins yet on how much longer they expect it to take to resolve all the issues with them, but we’ll update this post with more information as we get it. -dan w

Update Jan 29th, 2:15pm PST: We’ve been updated from our admins that FTP issues remain on a few shared hosting servers. Those are being worked on and should be resolved in the near future. The http services on dedicated servers have been resolved, but there are still reported issues with MySQL on those. A team of admins are currently working on resolving that. VPS servers were hit the hardest by this issue. Due to the number of VPS servers, it is taking our admins longer to go through and apply the fixes to them. Progress is being made at a quick pace to restore service to the remaining servers. We will update this post again shortly. -dan w

Update Jan 29th, 1:40pm PST: The shared end of this outage should be resolved at this time. Both http and ftp services are up and running on all shared hosting servers. Unfortunately, for the dedicated and VPS servers there are more specialized setups on those servers and it’s taking more manual intervention from our admins to get things back up and running again. Our entire admin team is focused on the remaining servers that remain down and are working on getting this up ASAP. More info to come shortly. -dan w

Update Jan 29th, 12:25pm PST: Http services on shared hosting services should all be back up and running at the moment. FTP services are still starting on shared servers and is more than halfway done, so FTP service should be restored everywhere shortly. Our admin teams continue to go through dedicated and VPS servers that are still reporting down. We’ll update this post again shortly. -dan w

Update Jan 29th, 11:55am PST: The http on most shared servers should be back up now. There’s only about 5 Apache’s on various shared hosting servers that are still having issues starting, but our admins are working on resolving those now. Our rolling FTP restarts are about half way through restoring FTP service to the affected servers. Our dedicated and VPS admin teams have been making progress on getting services on those servers restored as well, but we don’t have any hard stats on how much more work needs to be done on those yet. We’ll update this post when we have more information. -dan w

Update Jan 29th, 11:25am PST: We are making good progress on getting the affected servers back up and running. A good portion of the services that were having problems are now back up and running. Our admin team is working at full force to restore service to all the remaining services that are having issues. The team that was handing restoring shared hosting services is close to finishing apache (http) and FTP services updates/restarts. Our dedicated/VPS team are still working out a couple of kinks on the backend and will then be sweeping through and fixing any remaining down services there. More info to come soon. -dan w

Update Jan 29th, 10:45am PST: Not much to report since the last update. Our team of admins continues to work to bring up downed services. We’ll have more information shortly. -dan w

Update Jan 29th, 10:20am PST: Updates are still being rolled out across all the affected machines. We have a team of admins responding to services that are not coming up after updates and are trying to restore service to all affected services ASAP. We don’t know what caused the issues with the packages yet, and likely will not until service is restored everywhere and we can do a full autopsy. More info to come soon. -dan w

Update Jan 29th, 9:50am PST: We are still working on resolving all issues with the affected servers. This was caused by several software packages that were botched and/or removed by our packaging system. This was a new, unexpected bug that is unrelated to any of the outages earlier in the week. Our admin team is still pushing out rolling updates on all the related servers to get things fixed up. Many of you may notice your service restored already, but we are still working on the ones that are still reporting down. We will update this post again shortly. -dan w

Update Jan 29th, 9:25am PST: We are still working with our admin team now to resolve this issue. They are currently rolling out fixes across all the affected servers. Just a reminder, this outage is affecting FTP services on shared servers, FTP and web (apache) services on VPS and dedicated servers. We will update this post shortly with up to the minute information. -dan w

Update Jan 29th, 9:00am PST: We have located the source of the issue and our admin, dev, and data center operation teams are continuing to work together to bring things back up again. Thank you, again, for your continued patience. More updates to come. – Chalise

Update Jan 29th, 8:40am PST: We have located the source of the issue and our admin, dev, and data center operation teams are working together to bring things back up again. Thanks again for your continued patience; we will continue to provide regular updates as we have them. – Chalise

Update Jan 29th, 8:25am PST: Our admin team is hard at work to get this resolved as quickly as possible. We are finding this is also affecting SSH on some VPS and dedicated servers as well. Your patience is much appreciated as we continue working toward a resolution.  – Chalise

Update Jan 29th, 8:05am PST: Our admin team continues to work hard on fixing this so we can get your sites and services back up as quickly as possible. This is affecting FTP services on shared servers, FTP and web (apache) services on VPS and dedicated. Your patience is much appreciated as we continue working toward a resolution.  – Chalise

Update Jan 29th, 7:35am PST: Our admin team is continuing to work with the data center operation team to roll these updates out to VPS host machines; this also seems to be affecting dedicated servers as well, which are are currently looking into. – Chalise

Update Jan 29th, 7:10am PST: Our administration team is currently working with the data center operation team to roll these updates out to VPS host machines. – JJ G

Update Jan 29th, 6:40am PST: A fix is rolling out for the FTP services on the affected shared servers as well. – JJ G

Update Jan 29th, 6:20am PST: Our admins have identified the problem and are executing a fix across the affected servers. We’ll continue updating this post with more information as soon as we’re able. – JJ G

Update Jan 29th, 4:55am PST: No further information to pass on yet. This is still being actively worked on and our admins remain on the case. – JJ G

Update Jan 29th, 4:15am PST: The issue with ftp services also seems to be affecting some shared servers. – JJ G

Update Jan 29th, 4:03am PST: A configuration problem on the VPS servers has been identified and it is being fixed across the board for all web VPS customers. – JJ G

This entry was posted on Sunday, January 29th, 2012 at 3:35 am and is filed under Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

1,054 Responses to “Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down (RESOLVED)”

first. lol.

I’m sure you guys will have us up back soon enough. I have faith.

A day that ends in “y”!!

Another two hour outage. No reply or acknowledgement of the support ticket. No online chat. Documentation in the VPS help docs out of date. Wonderful.

“a subset of our VPS services are currently down” – how about some details like which “subset” – which servers, would that be so hard? When there is an outage like this we never know if the problem we are having is the same problem that they already know about. When I log into the panel, I want a big alert – YOUR A,B,C SITES ARE DOWN DUE TO KNOWN PROBLEM XYZ, otherwise how am I supposed to know if I should submit a ticket? dreamhost needs their own integrated monitoring and alert solution – I use a third party solution and there is no need to be woken up in the middle of the night like tonight if the problem is dreamhost and they know about it and there is nothing I can do!

it is not only vps machines, it is also on dedicated servers!

is that why my webmail is not working?

adam

Is there a larger problem going on here, such as DH not being able to keep up with growth or something? They were pretty solid 8 years ago when I first started out. Lately (couple of years) they’ve been getting spotty. Is there writing on the wall we should be reading here, such as what Snagglepus would say — exit stage left????

sites vps barriodelcarmen down…

I’ve been a happy customer for almost 8 years, but I fear it’s time to explore options.

Why not tell us what the problem is??? Is this why email suddenly doesn’t work and I can’t log in AGAIN? Or is this something else? I don’t know if I should be reporting the email issue or if this is the “subset” you’re working. A little REAL info would be helpful.

I keep having to up my VPS memory allocation and paying more and yet, getting more insufficient memory allocation messages by the week.

ok. At least I know why. btw, I have 23 days to make a decision on renewal. This should help me decide.

third major problem in a month? wtf, guys.

Other outage?

I have 10 days outage and lost many files due your “backup” issues …

Now we have this again .. for all VPSs?

WTF guys?

Chat is currently offline. If you have an account with us, please submit a support ticket under Support/Contact support in panel.dreamhost.com.

and what about dedicated servers? here the sites are down!

tell us when you will fix

@sesame Yeah, I started getting those memory allocation messages a couple months ago, even though my traffic was down. I don’t know how many times I’ve been down for more than an hour in the past two weeks, and when I needed it most the customer service chat was down.

I’m a long-time customer, but I have been looking for alternatives as others here have said. If migration were as simple as pushing a button, I would have been out of here a week ago.

your service is so poor…we pay you so much money for these vps and dedicated servers and every day there is a different problem which costs us loads of money. i’m going to move all my sites to some other provider asap

seems our server is not accesible from europe, been like that whole morning… pings okay, but ftp, http and mysql are not accessible…

4 major outages in the last 20 days … INCREDIBLE !!!

@Dude same problem here. Are you on VPS or dedicated?

I’m on a VPS. Been down all morning here in Europe. Was down for a couple of hours this time yesterday, and for quite a while last week! Was in the middle of an article at around midnight. Stayed up until 3.00 am to see if the problem may be rectified, but it wasn’t.

Everyone must laugh their socks off at people who use DH. It’s embarrassing – and not cheap!

Plz tell us when you will fix this.

@DreamHostSimon is the Twitter from DH CEO

https://twitter.com/#!/@DreamHostSimon

I think he is now drinking a beer while we lost money ..

“ECONNREFUSED – Connection refused by server”.

I’m done with these clowns, 5 year customer, paying this much for a service that breaks often without any notification. I have students unable to finish ther assignments.

As soon as they get their system up (? Today? Tomorrow? Next thursday?) I AM GONE.

Yes, I am also a bit over the problems, which in my mind are starting to get utterly ridiculous. I am asking my clients to prepare for a migration…

It is terrible. Every week there is a serious and important issue. In the previous issue the cackup was bad and many files did not rise correctly. I am still trying to know the files that failed of the back up. Today every down is again.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Is webmail down?

@Kevin Davidson You too? It’s a bit suspect isn’t it cos I find it ridiculous I’m paying like $80 – 90 for daily traffic of around 12k. I think it shouldn’t be that much. I’m also looking out for others but not sure what is good. You have any idea? Or anyone?

This happens way too often, folks.

January 29th, 2012 at 4:50 amAnthony Ramtulla Says:

This is not good. Very bad for business, I have egg on my face in the customers eyes.

lost money? need to change my host ASAP or not?

i suggest everybody move to other provider…dreamhost became dreamjoke….i’m going to share this thing with all the people i know so that they will never think for a second to sign up to your service.

I think we understand the meaning behind the name “Dreamhost”. Your servers seem more interested in sleeping and doing nothing than being awake and serving your clients.

A while back I signed up for a $3.95 a month shared hosting plan with one of your competitors to try them out and I’m really impressed with them. Their shared servers function better than my Dreamhost VPS that is costing me much more money, stress and downtime, so it looks like I will be moving all my hosting needs to them and I will no longer have to visit this site that I have now bookmarked because I have to visit so frequently. Soon I will be able to relax and sleep soundly, just like your servers. Sweet dreams.

I have opened a ticket about 3 hours ago and choose the ‘people are dieing..’ options. There was no response.

ps47237
2 hours 18 mins ago: Outage verified: We are actively looking into resolving it.
2 hours 22 mins ago: Outage first reported.

Status: Connection attempt failed with “ECONNREFUSED – Connection refused by server”.
Error: Could not connect to server

Your website does not appear in the Apache configuration file.
Your website does not appear in the Apache configuration file.
Your website does not appear in the Apache configuration file.
Your website does not appear in the Apache configuration file.

It was really funny! Just need support with phone and they add this:
Phone and Chat Support
$9.95 monthly

VPS mounthly $60 + UI $3.95 + Phone Sup. $9.95 and still waiting….

January 29th, 2012 at 5:09 amserver is down Says:

What’s happen! i am paying 50-60 usd/month for a VPS and all my webs are down! And this isnt the first time that happen

Does anyone have any suggestions for a better Web Hosting company ? I live in Australia but maintain a website for my sister who lives in Ireland. Any suggestions would be gratefully accepted. Can’t go on like this as we need daily modifications to the website, and some of our clients have actually missed out on castings because their CV’s and showreel links couldn’t be accessed because I couldn’t publish their updates – all because of the continual crashing of DH. I think it’s time we are compensated, as we are spending good money for a crap service and losing money.

please,

could you please just confirm that you are working also on the dedicated servers?
i’ve opened a urgent/blocking ticket: (#4712565 DOWNTIME on Dedicated server)and at the moment it’s 1h and 48 minutes and noone told me nothing.
at least a “we claimed the issue and we will let you know”….

I don’t know if this can help but on the support test link: https://panel.dreamhost.com/index.cgi?tree=support.test&amp;
it is telling me that: “Your website does not appear in the Apache configuration file.”.
I cannot neither login via ssh/sftp (Could not connect to server) and when i ping the domain it reply with “Request timeout” error on all of my dedicated server domain! (same for fps mysql) i’m in pain!

Does anyone else is having that kind of error?

please give me a feedback.

It’s now 8:10 AM and my site is still down.

I wish instead of teasing about better alternatives to Dreamhost, people posting here would actually post the names and URLs of other, more reliable hosting companies we could switch to.

I am tired of the constant downtime and the high rates of Dreamhost, I just don’t know where else to turn…

Have been using Dreamhost for my business in China. Have had to up memory and still had memory issues during times of virtually no traffic. This recent outage has pushed me over the edge; I’ve just made the jump to a local host here in Shanghai – bank transfer already sent. Sure, I have to apply to the Chinese government to have my business issued an Internet Content Provider license and have to abide by a ton of restrictive rules – but at least our customers will be able to access our site FFS!

just incredible

Heyy come on! I fell sad! Why i have upgrade to VPS… For more problem? My clients calling me every minutes :(

Really desapoiting. No response via email to my tickets, but I continued investigating and found out this post explaining the problem. But after 3 hours… finally a response to my email. Saying:

“We were unable to verify any server-wide problem related to
the outage you’re experiencing. If you submitted a support
ticket, we will be with you soon to help resolve this issue.”

How it is possible? Is it some joke? I’m paying a VPS for a domain + shared host for 2 other domains, all of them are down, and I have this kind of response?!?! That’s ridiculous!

I read that slicehost and linode are good but am not sure how much they charge.

After the constant problems I have been experiencing with my Dreamhost VPS, I turned to Certified Hosting (http://certifiedhosting.com/) to try them out and I have not only found them extremely affordable, their servers seem more reliable in my experience with them so far.

Two hours on a Sunday isn’t significant for the U.S. but other countries may be impacted severely. I’m less than a month using this server and may have to bite the bullet and move on if the next month sees several access/security issues. My clientele cannot be subject to outages.

January 29th, 2012 at 5:23 amserver is down Says:

I have a low traffic, about 1000 pageviews every day, however, for more security, i am paying 60 usd for a VPS, so is incredible that i am having these problems with DH, all my webs are down! This year, DH have been down 7 or 8 times!!!

I think that is time to look for an alternative…

Salutacions cordials des de Barcelona

There are plenty of “better” hosting companies out there — but be prepared to pay a lot more money. Take Rackspace, for example: they are a product-leading company, but they are astronomically more expensive than Dreamhost. Also, for the most part, Dreamhost has pretty fast customer support that you don’t always get from the bigger companies.

Look, Dreamhost is a price-leading hosting company. If you don’t like the quality and want to switch, be prepared to pay a lot more.

That being said — c’mon Dreamhost: get your VPS running already!!!

4 major outages in the last 20 days

I think the hack was more damaging than we thought. Be patient.

If you have got a VPS from DH you have got mail every week with this subject: Attention Required: Your DreamHost VPS needs help!

Even with two site on VPS. Your DreamHost VPS eat a lot of memory without any traffic. It has just exceeded the memory allocation that you’ve established. When you check the memory graph you never see more usage then allocated. Use DH with sharing hosting but never use with VPS. Its like nightmare…

Man….for the past almost 2 years now, I have had no issues with DH, but the last 2 months have been nothing but total headaches. THEN….I switch to a VPS thinking I will get more reliable and faster service and BAM….all my sites go down again for the 2nd time in the last 2 weeks. I sure hope this isn’t a trend with DH.

What has been rock solid for me in the past has recently become a mountain of support ticket requests.

I’m sure your webs SEO is getting better w this.

Wow, this is becoming somewhat of a regular occurrence guys…

seems to happen also with dedicated servers so it seems… as my sites and ftp are also down.

Guys… it is so frequent lately… No info, no live chat, no money back… Time to leave?

Come on guys, this is the third time this month I get my VPS down. I have no more excuses to say to my customers, what a hell are you doing?
Unfortunately, I’ll move all my sites to another company that be more serious and competent. That’s a disregard for your customers.

Actually, DH’s VPS eat little RAM compared to what you get elsewhere: here, your email and SQL traffic is handled on other servers, whereas other hosts put the whole thing on your VPS, that is on its RAM. So if you want to try it elsewhere, be careful with your RAM budget. Then the RAM is not as important as it seems for the VPS or server: there are other limitations which depend on the way your software behaves and those hurdles are not directly linked to RAM.

WTF – It’s to much for me.

I am losing money and traffic every minute.

100% I will move all my sites to other hosting now.

Is this a joke? A bad one… no feedback at all about the timeframe? WTF?

Update Jan 29th, 4:55am PST: No further information to pass on yet

I think they not know what really is going on

January 29th, 2012 at 5:51 amIncreasingly Frustrated Says:

Wow! In addition to today’s issue, ***SO*** many people here having the SAME issue I am having: Continual robot-email messages about how I need to INCREASE the memory (and cost) on my dedicated servers for a small handful of VERY low traffic sites.

It is time to move to a new home. Who has suggestions for alternative hosting companies? Anyone?

@sesame
I’ve been with a number of hosting companies over the years (7 years at DH) and they all have had severe problems at one time or another. It’s almost like trying to figure out which checkout line at the market is the best one. You switch only to find the other suddenly slows down.

There’s no company I would recommend, but the company I’m currently reviewing is vps.net. Lunarpages has also been pretty stable of late, but I’ve never tried to use any volume with them, just shared hosting.

January 29th, 2012 at 5:54 amIncreasingly Frustrated Says:

And, how freakin’ annoying is it that I spend a bunch of time in the dreamhost panel trying to figure out what is going on, performing a restart of both db and web servers, still having no luck, and THEN – and ONLY THEN – seeing their notice about their own issues on the page you have to use to contact support.

Downtime + Completely Wasting My Time

F-U Dreamhost.

I have both shared hosting and a VPS on Dreamhost. The shared (cheap) site has been doing much better than the VPS in all this mess.

UNBELIEVABLE!!

Does it really take 4 hours to fix an issue on a Sunday? An issue that you supposedly have the answer to? Then ALL of my passwords had to get changed EVEN though I dont use FTP but use SSH? C’mon Dreamhost! I have been a customer for over five years and I understand things happen. If you are having this many problems in January then maybe its time to seriously consider upgrading your platform to deal with these issues.

Well, there’s nothing left for us but group sodomy. I’ll start.

big event sponsored by google is going on at krk.org.i/athena . you guys are making 2000 participants wait for a long time . for gods sake please solve !

January 29th, 2012 at 6:04 amIncreasingly Frustrated Says:

I want to have my sites on the same server/configuration these dolts use for Dreamhost.com! Their main site is ALWAYS up and running well whenever all of us our down. I wonder who dreamhost uses to host their own site!

Our adminstrative site is down again. I thought upgrading to VPS would be better and more stable but it seems I was wrong. I hope the site comes up soon.

c’mon DREAMHOST more than one hour without any information about the issue. You are kiding with us customers. I think it’s better change your staff to clean this mess, we all are tired this constantly outages.

After nearly 8 years with Dreamhost it’s time to find another option.

I preffer ftp service down than a running one with overwrite permissions denied, is less frustrating..

thank you system administration and data center operation teams!

btw, you guys need help? just rememeber to RTFM before touch a machine.

Does anyone ever been credited according to their 100% downtime guarantee here?
http://dreamhost.com/web-hosting/

I never remember being credited, although there were numerous downtimes and I’ve created a ticket as soon as I got a message from my monitoring system.

Is this just marketing BS or they actually do that?

Hmm, I didn’t notice that there was an outage, so I scheduled a reboot. Now I can’t even ping my machine. Good luck, I really hope you resolve this soon.

I can’t acces my FTP, please solve this problem, I want to send big file to my friend.

Waiter, check please!

Waiting for good news and still trust DH. Please solve this issue today.

anyone wanna recommend a better host? because I cannot deal with this shit any longer. most people make money from their websites and this constant downtime isn’t helping our pockets. i am beyond beyond beyond tired of waking up just to see that my site has been down all nite and all morning.

4 hours of downtime … we will get 4 days free

You’re all frustrated.

I know. Your sites are down, you haven’t heard enough, we have no idea what’s going on.

Let’s channel all this frustrating into something more productive: sexuality.

I’m still pre-op and have a prostate. Would anyone like to pound it flat with an erect phallus? Or since I’m pre-op, I can do the same to you.

Please provide more info regarding the issue. I need to have something to say to my customers. I expect some kind of compensation.

I subscribed for shared hosting last week, the host ps80233 has been down for the past many hours now. Totally disappointed, someone please suggest a more reliable shared hosting for wordpress even if its costlier.

I think VPS and Dedicated here is the same as shared.

If I was really over a VPS, an issue at server A, can´t damage my server at B location.

For me this is a biggest lier … DH Fails

By the way, the talk about 100% uptime guarantee is misunderstood. Here’s the contract you actually signed:

“# DreamHost Web Hosting makes no warranties or representations of any kind, whether expressed or implied for the service it is providing. DreamHost Web Hosting also disclaims any warranty of merchantability or fitness for a particular purpose and will not be responsible for any damages that may be suffered by the Customer, including loss of data resulting from delays, non-deliveries or service interruptions by any cause or errors or omissions of the Customer. Use of any information obtained by way of DreamHost Web Hosting is at the Customer’s own risk, and DreamHost Web Hosting specifically denies any responsibility for the accuracy or quality of information obtained through its services. Connection speed represents the speed of a connection to and do not represent guarantees of available end to end bandwidth. DreamHost Web Hosting expressly limits its damages to the Customer for any non-accessibility time or other down time to the pro-rata monthly charge during the system unavailability.
# DreamHost Web Hosting specifically denies any responsibilities for any damages arising as a consequence of such unavailability. In the event that this material is not “Server-ready”, DreamHost Web Hosting may, at its option and at any time, reject this material, including but not limited to after it has been put on DreamHost Web Hosting’s Server. DreamHost Web Hosting agrees to notify Customer immediately of its refusal of the material and afford Customer the opportunity to amend or modify the material to satisfy the needs and/or requirements of DreamHost Web Hosting. If the Customer fails to modify the material, as directed by DreamHost Web Hosting, within a reasonable period of time, which shall be determined between the parties themselves, the contract shall be deemed to be terminated.”

http://dreamhost.com/terms-of-service/

“DreamHost is under no obligation to compensate Customer for downtime, whether the downtime be caused by Customer, DreamHost Web Hosting, or DreamHost’s upstream providers.”

And:

“# Neither DreamHost Web Hosting, its employees, affiliates, agents, third party information providers, merchants licensers or the like, warrant that DreamHost Web Hosting’s Server service will not be interrupted or error free; nor do they make any warranty as to the results that may be obtained from the use of the Server service or as to the accuracy, reliability or content of any information service or merchandise contained in or provided through the DreamHost Web Hosting Server service, unless otherwise expressly stated in this contract.
# Under no circumstances, including negligence, shall DreamHost Web Hosting, its offices, agents or any one else involved in creating, producing or distributing DreamHost Web Hosting’s Server service be liable for any direct, indirect, incidental, special or consequential damages that result from the use of or inability to use the DreamHost Web Hosting Server service; or that results from mistakes, omissions, interruptions, deletion of files, errors, defects, delays in operation, or transmission or any failure of performance, whether or not limited to acts of God, communication failure, theft, destruction or unauthorized access to DreamHost Web Hosting’s records, programs or services. Customer hereby acknowledges that this paragraph shall apply to all content on DreamHost Web Hosting’s Server service.”

Good luck demanding your four free days.

Ruben

I have never ever been credited with downtime and have suffered this several times. If it smells like bullshit, just hope you don’t step in it.

I suggest a lot if twitter shaming to @dreamhost is it any wonder other web host outfits mine that stream for potential customers?

I’ve been close to moving before but am absolutely done.

I agree with @Rafael, a VPS server (like mine) should be a proper private server, not a shared one. I had a shared service a few months ago which I have supposedly upgraded to VPS. However the only noticeable difference of VPS is the bill.

this sucks ! And as many said before its not the first time…We all pay good money for our VPS and nobody from DH is able to tell us wtf is going on and when this will be solved…

Thank you for the information.

It always amuses me to see all the whiny little babies here. Check the status page (Which you did if you posted) and subscribe to the RSS feed. Try not to inundate them with tickets for a known issue while they work the problem. Quit whining, nothing is ever perfect when it comes to hosting. I am satisfied with the uptime of my sites and know that there will be problems occasionally.

V=Virtual Private Server. You want a dedicated private server, pay for it. Better yet, take your whiny drama troll butt to the store, pick up a server, get a static IP and host it yourself.

Trolls and drama queens.

-Happy LONG TIME Dreamhost customer.

LOL Dreamhost. VPS sites down, and now for over 3 hours. And I thought I had it bad with the slow shared servers.

I’m definitely leaving.

I don’t have a VPS but my WordPress installation on shared server is SO SLOW, like 6 to 10 seconds Time to First Byte even on an almost NAKED PAGE. Bad, slow old servers.

Yoast.com (the plugin developer Joost) recommends WestHost if you’re shared server, or sister company VPS.NET for bigger needs.
http://yoast.com/articles/wordpress-hosting/

I hope you all leave Dreamhost.

I’m SO glad that we left DH. Now I can read this without any stress.

So I guess we’ll have all no less than 3 days free?

“DreamHost has a solid commitment to providing you with the very best web hosting service we can – that’s why we can offer you a straightforward and rock-solid 100% Uptime Guarantee!
How it Works
If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that! So… sign up today! “

oh yeah “limitless possibilities” Get unlimited downtime http://dreamhost.com/web-hosting/

It’s not just Apace and http://FTP... I run a server software (Unity Asset Server)and it’s down as well :( WTF!

I can’t log in to my FTP account from past 4 hours. When will you solve problem?

There are other services too that are affected: WEBMAIL

Can’t login, even after password change. WTF!

January 29th, 2012 at 6:53 amIncreasingly Frustrated Says:

y0himba, clearly you are without a clue about web hosting or even the simple aspects of running a business. Go right ahead and continue to be satisfied with crap. The rest of us want value for our dollars and truth in advertising.

“If your web site, databases, email, FTP, SSH or webmail is unusable as a result of a failure in our systems and for reasons other than previously announced scheduled maintenance, coding or configuration errors on your part, we’ll credit your next invoice with 1 day hosting free for each 1 hour (or fraction thereof) of service interruption; up to 10% of your next pre-paid hosting renewal fee. We begin assessing credit from the time you open a support ticket to report the problem. That’s all there is to it – no arguments, no hassle, no pages of fine print – DreamHost’s 100% Uptime Guarantee – it just doesn’t get any better than that!”

That’s not in the contract, but maybe they’ll be cool about it.

On which page did you find this?

@Increasingly Frustrated:

It could just be blown out of proportion because having your site down is incredibly frustrating period. Most webhosts are incompetent. Dreamhost may be a little less incompetent than most, and have nicer policies.

But I really came here to talk about prostates and nipples.

right said “Increasingly Frustrated”

No update in over 3 hours… can you at least let us know what is going on?

and another thing. These emails about us exceeding our resources and needing to increase our capacity. Slide the bar up, fellas. Do you go look at your chart after you get an email? I have every time and not once was the spike recorded. Here is my conversation with customer service last month.

General Info
Chat start time Dec 26, 2011 6:00:09 PM EST
Chat end time Dec 26, 2011 6:20:16 PM EST
Duration (actual chatting time) 00:20:06
Operator Jeremy T.
Chat Transcript
info: Please wait for a site operator to respond.
info: You are now chatting with ‘Jeremy T.’
Jeremy T.: hi there! how can I hepl you?
richard: I have a question. Why do I keep getting emails telling me I’ve used up my allocated memory on the server and when I go to look at the graphs..I am not even CLOSE to uising that much memory..
Jeremy T.: The graphs can’t show it when you hit the limit as they only update every few minutes.
richard: I have 374MB and I have never exceeded 200mb
richard: but I go back and check..and I have NOT ONCE exceeded 300mb and I am allocated 374mb
richard: Im concerned about this
Jeremy T.: The graph only checks once, every few minutes. It isn’t capable of showing a spike that takes you over your memory limit.
richard: but wouldnt I see the spike at some point?
richard: look at my graphs..
Jeremy T.: The graphs can’t show a spike. That’s the entire reason why we email you when it happens.
richard: there is no spike is what Im trying to explaint to you.
richard: the spike is well under..100mb under my limit
Jeremy T.: I know it doesn’t seem like there was, but as I was saying, the graph only checks the server once every few minutes. Between those intervals, your memory usage is unrecorded. That’s when you would have hit your limit.
richard: if that is fact, then why wouldnt I see the spike in my history at some point?
Jeremy T.: Because it would have happened in between graph points, when your server isn’t checked.
richard: so you have a graph that does not show the peak points?
Jeremy T.: Unfortunately, that’s correct. THat’s the reason why we send you an email when you hit your limit, so you can know when exactly this happens.
richard: therefore the graph is useless.
richard: because ti does not accurately show my peak usage
Jeremy T.: I’m afraid that it is not useful for troubleshooting memory spikes.
richard: so your email points me to a graph that does not show the peak that exceeds my limit, correct?
Jeremy T.: Correct, as that is the page where you can increase your memory allocation.
richard: This is a serious flaw and I cant but wonder if it is an intentional sales gimmick.. You provide no tangible proof that I have actually exceeded my limit. I may have to consider moving to a different host when my time expires in February unless I can get a better result, Jeremy.
Jeremy T.: I’m afraid that we can’t troubleshoot your memory usage for you, but you can follow the steps in these articles to get an idea of what’s going on:
Jeremy T.: http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage
Jeremy T.: http://wiki.dreamhost.com/PS_Optimization
Jeremy T.: http://wiki.dreamhost.com/DreamHost_PS_Troubleshooting
richard: nope..already done it. If I have exceeded my limit..SHOW ME the data
richard: I wuld like to know what my peak usage was
Jeremy T.: Your VPS is killed as soon as it hits the limit.
Jeremy T.: So your peak usage is whatever you currently have allocated.
Jeremy T.: As soon as it hits that mark, it gets restarted and you get sent that email.
richard: so you can provide a graph showing when I hit that limit?
Jeremy T.: Not currently. That is why we send you the email.
richard: ok..I am going to file a ticket/complaint over this. I want to know the exact time I hit my limit. The server records the reboot and I will need that time to compare against my posts.
info: Your chat transcript will be sent to fishfirewrite@gmail.com at the end of your chat.
richard: thank you for your time, Jeremy.
Jeremy T.: Have a good day, then!

I HAVE a VPS but still all my sites are down for about 5 hours now…

There was an update 20 minutes ago. It’s listed in Pacific time.

Thanks for the update DreamHost. It’s unfortunate that you’ve had such downtime lately, and this is getting a little out of hand.

I would hope you refund everyone at LEAST this day’s payment (i.e. our vps charges for the day). Apart from RAM issues and this recent hacking, I’ve had nothing but good things, and support has always been great for me. Just keep up your security and we’ll be happy to stay with you.

It did again. My domain can be pinged, but can not be accessed through browser.
I WONDER why dreamhost often down these recent years?

I have been dreamhost user since 2007, never seen these such often downtimes before.
Hopefully dreamhost more stable from now on.

Thanks for the update that you haven’t resolved the issues after 3 hours of downtime. That 100% uptime promise reminds me more of the old saying about 2+2=5.

Changing for a new domain right now..

Is This a Class Action Lawsuit ????

been down too long… Fix it! This is your business. And please add more Sunday AM staff; most of the English-reading world has been up for hours already, have checked their morning news and gone out for the day.

@Dreamhost people: NOT only web and ftp services are down! I’m running a custom SCC server on my VPS and that is not working either!!! You have a more extessive issue at hand, not just what you are outlining in your darn post.

No comments about the DH Services.

is increasingly difficult to continue with the DH. Instead of helping us grow our plans to increase esles us down even more.

If anyone has any indication of providers that work share.

Email and webmail down. I don’t have a VPS service, just plain old vanilla. After the hacking last week and new passwords being reverted to old ones, confidence shaken again (at least the messages here aren’t in the “cute” teenager DH style).

@Tat: I’m starting to see the basis to establish a class action suit indeed

Everything is down on all my web sites returning this error :

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

and now I reported the problem, they sent me this email :

We were unable to verify any server-wide problem related to
the outage you’re experiencing. If you submitted a support
ticket, we will be with you soon to help resolve this issue.

We’re very sorry for any inconveniences,
The UnHappy DreamHost Notification Team

İs this a joke?

I was already in the middle of moving all of my business over to HostGator. In the middle of moving 30+ websites, this happens. Again.

Well, at least I know I’m headed in the right direction. I’d really love to know some information right about now. Where’s your line of communication, people?

Guys!

I spoke to one of your support guys on Chat the other day, I think what he said is worth quoting here. I asked him about the risk of major downtime and his reply was, “We are not some mom and pop shop!”

This is a transcript of the relevant part of the conversation:

[04:09:59] ChrisL: No, but seriously, we’re not a mom and pop shop
[04:10:12] Andy: So are iWeb a Mom and Pop shop?
[04:10:13] ChrisL: It would take armageddon in Los Angeles to bring us down

Note to ChrisL. My VPS has been down for many hours. So has LA been hit by Armageddon?

@Tat – no it’s not. get over it, it’s just the internet

3 hours of downtime is not tolerable. Makes yah wonder how competent their administrators are after a saturday night. Hmm..

Even SSH access is unavailable this is unacceptable. I’m just glad I am still in development stages on my websites and not live yet. If this continues I will seriously consider switching to somewhere like VPS.net.

No sales today! I thought… huunnmm let me check it, probably the web site isn’t working, because the dreamhost’s service down…. and “bingo” I found the reason. The biggest trouble is losing money every second that passes … and no one support is online to give us the reason … I want to know when it’ll be solved …

@Ulas: at least you got an answer :(

@IRFS the test gives the same result to me….. I thought it was only my site…..hope this will be fixed soon, I too have students waiting to deliver their assignments…. :-((

@Andy:

So has LA been hit by Armageddon?

I think all of Los Angeles wants this to happen.

@the rest of you:

Quit whining.

Seriously, I know this hurts. But you’re not as important as you think you are, and you’re not making huge assloads of money or you’d have a more bulletproof hosting solution. Further, there’s not much that can be done at this point. If you want to change hosting, fine, but quit posturing.

Do you know what really makes webhosting expensive? Dealing with irrational consumers. One of the biggest costs is technical support, which is usually hand-holding irrational people through simple procedures they could have figured out through google. Another is cleaning up the messes made by the clueless and their hacked together websites and insecure software configurations.

I think it’s great that I can come here and make degenerate comments about tranny sex and these don’t get noticed because everyone else is in such a hurry to complain about how their massive important websites are down and thus they will never become billionaires, as if that was what was happening. You know what that smells of? Pretense.

Knock it off, you whiners. The problem is being fixed. There’s not much anyone can do about that. If you want to go to another host after this, great, but right now you’re just generating 100+ whiny messages that are helping convince the Chinese and Russians that Americans are soft-ass whiners who will be easily defeated in a land war.

Chat is currently offline. If you have an account with us, please submit a support ticket under Support/Contact support in panel.dreamhost.com. Otherwise, please submit your sales requests at http://www.dreamhost.com/contact.cgi We apologize for the inconvenience.

Webmail not working, won’t accept username/password. DH has really gone to shit lately.

ps80127: 4 hours 20 mins ago: Outage verified: We are actively looking into resolving it. 4 hours 48 mins ago: Outage first reported.

Here, read this. This is someone with real problems:

http://www.cnn.com/2012/01/19/justice/diguglielmo-new-york-deli-killing/index.html?iref=obinsite

Now, what were you whining about again? Right, I thought so.

This is completely unacceptable. I was down for a week with missing data that was NEVER restored when dorbo was brought back on line (taking numerous days with the “wait to see what is on the backups so it’s not overwritten), then, there was a hack to the FTP system which you all changed my password, I changed it again, and you slicked that one out by rerunning the script, and now THIS?

I understand computers have issues (I work with them every day), but this is the 3rd, perhaps 4th major time that Dreamhost has gone FUBAR, and we’re not even 1/12 of the way through the year.

I don’t make money off my websites, so you’re not hurting my bottom line. But I’m also not getting the services that I pay YOU for. 7 years of being a paying customer and I’m not sure I’ll make it to year 8. Not unless service (and support) improves DRAMATICALLY.

4 hours 26 mins ago: Outage verified: We are actively looking into resolving it.

4 hours 53 mins ago: Outage first reported.

DH is my only service so far, have been happy with it, recently switched to my VPS. But things have been a little bit annoying lately, lot of down times, my sites have been hacked 2 weeks ago, DH system has been hacked a week ago and now on a Sunday all my sites are down again.

I am not yet planning to change my hosting but I am really annoyed.

4 hours 27 mins ago: Outage verified: We are actively looking into resolving it.

4 hours 55 mins ago: Outage first reported.

@Ron, you obviously do not run a site that actually makes money, Of course things happen but There is such a thing as accountability, but if you have no hits to lose, then you wouldn’t really care now would you??

@Random Emosexual: Ever considered Anger Management? Seriously, you would benefit from it! :-)

All Random Emosexual is doing is promoting a sex site by posting (with the link in his name). Please do not feed the troll.

JJ G – thanks for the updates!

Tat,

If you run a site that makes money, maybe you should be investing in your own server not using “Dreamhost”.

And we have a winner! 5 full hours of downtime!!!!!

4 hours 32 mins ago: Outage verified: We are actively looking into resolving it.

5 hours 0 mins ago: Outage first reported.

I joined DH in 2004 and this is the only provider I have been using but enough is enough.
I’m joining the “Ridiculous” and moving my 20+ domains to HostGator.

4 hours 37 mins ago: Outage verified: We are actively looking into resolving it.

4 hours 54 mins ago: Outage first reported.

@Andy:

I’ll go to anger management if you go with me. But I don’t feel like I’m the angry one here. In fact, your comment is ludicrous. We have 139 comments in this thread, most of them doing one or more of the following:

(1) Expressing outrage
(2) Questioning the competence or honesty of Dreamhost
(3) Threatening to move to other webhosts
(4) Raging about “losses” from their food blogs being down

Who’s angry here?

I tell you what, Andy… let’s wrestle. When you win (I am certain you will) you can have your way with me. Just tell me if you’re going to be Andy the woman, or Andy the man, and I’ll prepare accordingly.

@M. Arif:

I’m joining the “Ridiculous” and moving my 20+ domains to HostGator.

Yes, that is indeed ridiculous.

@Jason:

If you run a site that makes money, maybe you should be investing in your own server not using “Dreamhost”.

It’s a sane one. How did this happen?

I think $20-60 per month is no trivial sum, but if my business were wholly dependent on it, I’d aim for some more secured hosting. The fact is that screwups happen at any price level and generally they get well-hidden, but until you pay for true enterprise hosting, expect errors.

4 hours 39 mins ago: Outage verified: We are actively looking into resolving it.

5 hours 7 mins ago: Outage first reported.

Update Jan 29th, 6:20am PST: Our admins have identified the problem and are executing a fix across the affected servers.

Its over 1.5 hours, how long does it take to upload a fix?

@jason yup, you are correct, Thank for the input, but in the meantime, I need these guys back up to move my stuff

I think everyone that’s saying its “no big deal” works for DH!!

4 hours 43 mins ago: Outage verified: We are actively looking into resolving it.

5 hours 10 mins ago: Outage first reported.

Sure, this downtime isn’t the end of the world.

But I just signed a new contract with an ad agency to run their ads based on my blog’s hit count, and this downtime will destroy my hit count and will cost me that contract.

So yeah, basically all the money and the years I’ve invested into making my blog pay off have been ruined because of this downtime, right here, right now.

Am I going to switch to a new host? Fuck yeah.

Our site has lost over a 100K page views by now, that is Not “No big deal”, in case you don’t understand a double negative, that IS a Big Deal

crontab is broken.

maybe this incident is related with the security issue of the other day?

4 hours 52 mins ago: Outage verified: We are actively looking into resolving it.

5 hours 20 mins ago: Outage first reported.

Guys it’s not funny! My sites are down several hours. How can you guys take money for down websites? First you guys had hacking issue now the server is down since 3:30AM and Who is going to pay for my lost business?

All my websites are getting:
***********
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
***********

Do something about it ASAP!!!!!!!!!! I thought VPS is like I have my own dedicated server, why are you keep bringing it down?

Your last update was an hour ago. When is my website going to be online?

My web site has been getting nearly one million visits a day. Yet again, it’s down. When are the problems going to stop?!?!

It’s not just web services (ssh / ftp) it’s everything that’s down… ssh, ftp, web, etc…

I started DH with shared server, ıt was down all the time, they told me to pay more and get a VPS. They told me that it was going to be my dedicated server. Now it has been down for at least 5 hours now. I run a web site which makes money and this is going to far. I am very frustrated. I give another 3 hours to dreamhost. If thing don’t turn around, I will have to make a decision.

Can someone give me some options as to other hosting services and the moving process? I have not done it.

By the way, some people posting comments here appear to be minimizing the issue. They are either working for DH or they have no idea what it is like to take people’s money for a service and remain accountable for it.

ps26662 is part of subset

Hi

As well as losing lots of money today it is still costing me on Adwords when people click through!

OK, let me ask, which other hosting company would you all recommend?

If we contact them collectively as disgruntled DH customers they will discount their offering to steal that much business.

@Mikey

Your web dont have a million of visits every day, you are just a lier :-)

THIS IS ABSURD! I think Dreamhost need to be upfront about what the problem REALLY is. This doesn’t look like a common update issue. If there’s data loss, SAY THAT THERE IS DATA LOSS! Tell us something so that we may begin looking for solutions ourselves, because obviously yours are not working!

This was definitely the final straw. Last year there was two instances where Dreamhost went down two or three times in one week. In 2012, they wasted no time in making sure it happened yet again. I’m letting go of their services the minute these sites go back up and I find a viable alternative.

@Tat

Try running a business that is actually profitable. Losing a 1/2 days worth of work is nothing if you are making money the other days. When my site comes online you should buy my book, it will show you how not to suck at life.

Jan 3: Network outage
Jan 13: Network outage
Jan 18: Major outage
Now this? What is going on with Dreamhost? They used to be reliable, but seem to go down on a weekly basis now.

@Mikey
“Your web dont have a million of visits every day, you are just a lier”

Does that really matter? Let us say that he is making 10000 hits. Is this the issue here? Are you having problems processing information? Or are you doing this intentionally?

@ron

If you’re website is “profitable”, how and can you justifiably imply that it is acceptable to lose a half-day of uptime? For those of us in the travel industry, it’s CRIPPLING to lose a Sunday morning of visits, and I assure you, that my sites are VERY profitable.

2nd downtime in 4 months for me… and I needed that site up today, customer is telling me to start looking at AWS.

Boy oh boy! It doesn’t take long for people to leap on here and try sell us stuff.

Ron: Shame on you!!!!

and as for Random Emosexual: I’m surprised you have a hand free to type with!

@y0himba – You bring up such a good point. Wow, how insightful. I cant believe I didnt think of that.

However, maybe I did. Maybe I have 8 servers that I use for hosting. Maybe I use Hostgator for offset hosting. Maybe I use Dreamhost for some extra offsite hosting. Maybe I use BlueHost for even more offsite hosting. Actually, I do all of those things.

When it comes down to it, its business. Downtime is downtime. I dont pay for downtime; I pay for uptime. Yes, issue happen but to continuously happen for a month is ridiculous. Some of the complaints on here are just people venting and I can understand what you are saying concerning that. However, for others who depend on Dreamhost, where they are hosting their livelihoods and losing money due to downtime, its not quite the same.

Even ONE site being down increases my tech support calls. Then my tech support my expenses go up. Its all relative.

Dear Virtual Server, Get well soon :-)

Who gives the best hosting in the US?

Can someone name some of them?

I’m pretty sure I didn’t sign up for a christian server. In that it should also work on Sunday.

Anyway, I hope you resolve the root so you don’t need to have these headaches any more :/

@diana I think our VPS needs a defibrillator!

@y0himba You are simply unhinged! Are you seriously telling me that you are happy with 5+ hours of downtime?

I’m not like the other folks who’s incomes hang in the balance over this downtime, but I must say this lack of communication on Dreamhost’s part is not what I expected.

First of all, I had to learn my sites were down from a reader over Twitter. Not via an email from Dreamhost.

Secondly, there’s little to know information here on the status of the machines affected.

Thirdly, I still don’t know if my VPS is one of the ones affected or if it’s just another RAM problem, as I still have not heard back rom support. However, I’m guessing I’m affected as bumping my RAM and restarting my server did not alleviate the issue.

How could Dreamhost make this less awful? Email the affected web masters as soon as a problem is noticed. Some of us have people we need to tell, and we’d rather contact them before they notice the issue. Respond quickly email tickets from web masters asking if they’re affected. Yes, it’s the weekend, but surely emergencies necessitate paying support staff a little overtime. Lastly, do better at informing us than this rather cryptic blog post. Are we coming? Are we going? Can’t tell, and you are not reassuring us.

I’m not angry at you, Dreamhost, but I hope you’ll take some lessons away from this.

@Ron

lol, Only reason I have time to respond to ignorance is because DH is Down. Now I’m trying to figure out, do you work for them, or are you the Hacker that brought them down? You are too ‘good’ with this. With that being said, I will not respond to you any more because even with all this free time, I’m too bored with you -_-
I swore the Circus left town, guess they left you behind.

Victor Pimentel: que pequeño es el mundo… internet.

@Hershal if your site is profitable enough to lose a day of downtime and still feed your family, why require anything else? Only a greedy capitalist would need even MORE money

Just saw your comment about this not impacting mail or mysql, if that is true you have bigger problems because my site has not been able to connect to the mysql server since this started. Can’t ssh to my server either, only getting connection refused. Can’t even run mysql from the command line on the ssh connection I left open from last night, only get mysql “No such file or directory”

8 of my websites are down for more than 6 hours. My clients are pissed. What other hosting service would you recommend?

Dear oh dear! It seems that @Ron continues to be an idiot!

@Ron

Are you working for DH? (better possibility)

Whoever tries to defend 6 hours of “charged outage” either has cognitive impairment or works for DH.

Can someone tell me a better host? HostGator? do they have a VPS dashboard like DH does? I am getting really sick of this.

I DO NOT WORK FOR DREAMHOST ; however, dreamhost continues to provide quality uptime for the majority of it’s customers. We are but a small percentage of users affected by this outage.

lol, Yeah he works from them, Everybody just stop giving him the time of day, and lets collectively work on a real solution

Earth to @Ron Could you log off!

I think your browsers are failing to render this page properly. I stated I do not work for Dreamhost. This outage has been good for me. I have had the opportunity to go outside and play catch with my son and spend quality time with my wife, rather than sit in my office and code / make money all day.

Ron is clearly a terrible businessperson…and apparently a socialist? Well, we’re definitely sharing this loss of productivity from this shared downtime of our shared hosting, so socialism prevails!

@Ron, why would you bring politics into this issue of clear incompetence on the part of Dreamhost? If you want to continue paying them for terrible service, go right ahead. Allow me to reiterate: DOWNTIME ON A SUNDAY MORNING IS CRIPPLING in the travel industry. Dreamhost has basically shat my week away.

@Ron I wish you were still outside playing catch with your Son!

*Update Jan 29th, 8:25am PST: Our admin team is hard at work to get this resolved as quickly as possible. We are finding this is also affecting SSH on some VPS and dedicated servers as well. Your patience is much appreciated as we continue working toward a resolution. – Chalise

lol, This is turning into a comedy show, this guy thinks we are as slow as DH’s servers, when have you done these things when your chatting traded things on this forum?? Come DH Employee of the Month!! Yes that is to “Barnum & Baily” Ex Employee @Ron

Chalise ~ how about you get the sites up first. then worry about FTP and SSH. BTW, VPS’s should be worked on first since we pay more money that shared hosting accounts. I am tired of paying $50 a month and getting downtime like this. PLEASE GET OUR VPS’s UP.

The only thing @Chalise did not tell us on the latest update was how long! As well as losing money today I wanted to do some updates to my sites and had a client site to complete.

Thank god I have a WAMP server setup and the latest Backupbuddy versions to look through!

Dreamhost – Not good! AGAIN! Not good!

You promise much and deliver less!

@Brian I used to use Rackspace back in the day, and they never gave me a hard time about anything. Come to think of it, I can’t really remember why I switched. Dreamhost seemed like the more cost effective option, I suppose, but you get what you pay for.

If you are looking for a shared hosting option, I also know that Yahoo! provides some good cloud hosting, and unlike Dreamhost, Yahoo ain’t goin’ anywhere. Same goes for GoDaddy. I think price-wise, they are about the same.

Ron, go back outside and play with your imaginary family and make sweet love to your imaginary wife. Clearly the rest of us as angry and we have a right to be. Make haste.

Ron,

My browser clearly renders my page okay. It is so clear that you not only work for DH, but retarded enough to make it so overt.

BTW, if the majority of DH user are okay, let them continue their services, the minority needs their money back for the downtime and loss of business. Ron, I understand you are impaired but you should be able to process as least this basic statement.

lol @ all you guys. I’m going to take my family out on the yacht, when I get back I am sure everything will be back up.

@Ron – quite simply, we are paying for a service. When that service is consistently going down it is hindering most of us our livelihoods. If you are well off enough to be able to deal with these issues then that’s great for you. Please however do not try to pass off this poor service as something positive.

It’s nice that you get to spend quality time with your family. You clearly chose the right host to inconvenience you enough to be a dedicated family man.

Now please – go back outside, some of us have work to do.

I guess it’s not so bad for you guys in California but for us in England who run B2C businesses (@Ron: That stands for Business to Customer) we have just lost a whole trading day!

I expect DH to read this message and let me know what they intend to do commercially about it!

Once they have all of our servers live that is!

My behavior is quite odd. I get shell, sftp, mysql, and .html and robots.txt work. But all .php files fail. Even the simplest test.php file.

I only run my personal portfolio from DreamHost, so as long as others can see it online I’m not too upset, but not only did I renew my hosting for another year today, I finally got off my butt to update my site and now I can’t upload it. It’s a minor frustration compared to what others are experiencing, though Hoping the recent rash of issues clears up soon.

I have also been having a problem with VPS reboots due to excessive memory. This has been surprising to me, because my web hosting patterns have not changed, yet suddenly my machines are rebooting every few days. I’ve been looking at using a different web server, hoping that would solve the problem. Apparently, though, the problems are deeper.

I have everything down ! Not a single piece is working, FTP, Shell, HTTP, mysql…

FTP says :
The remote host actively refused the attempt to connect to it.
1) Please verify that the destination host name or IP address is correct.
2) Verify that the connection port number is correct (under Site Settings | Type tab).
3) The remote server may be temporarily or permanently inaccessible (try again later).
4) Verify that you have chosen the right protocol (SSH2, SSL, FTP, etc.) and have setup all required options for that protocol.

Http says:
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

DH says:
There has been an update on the service interruption
you have asked to be notified about:

Outage resolved: No server-wide problem was found.
(2012-01-29 07:14:08 PDT, 0 secs ago)

January 29th, 2012 at 9:03 amGet Rid Of Simon Says:

Stuff like this happened a lot less before Simon became CEO. Get rid of Simon.

I think Ron is DH’s own Version of Siri, or theSim Simi App, with automatic tarded responses

I wonder if we will get a discount code or something

I have 2 millions visits every day

@Ron Yes but they are little fairies and pixies fetching your happy pills!

I hope our files, databases are still somewhere and they will be back. I don’t want to lose them :(

My site’s been down for over 10 hours. When my subscribers click on their email updates they (obviously) just get a blank page. Genuinely cannot believe how poor this service is.

Also can’t believe the people saying we should expect poor service because the price isn’t as much as other hosting companies.

If they can’t provide the service, they shouldn’t be trading.

Hey! @Ron :) I want to visit ‘you’ every day and i think i can found 2 millions with me who want to visit ‘you’ :)))))

LOL, Good One @Andy

@ron i have 800 million visits a minute. beat that.

@Ron – is dreamhoststatus.com your website? – since they clearly average 2 million visitors a day.

I’ve my site down. From hours. I can’t access mysql for backup or ftp. Down. All sites depending on my sites are slooow.

I do appreciate the updates, but a timeframe is what most of us want, PLEASE

Hello,

When is this going to get fixed? This has been happening too many times. I have several sites that are down and losing money.

Dreamhost, this is your last chance. If it’s not fixed in the next hour, I will be looking at new companies to switch my business.

This is unacceptable and outrageous.

@Chalise of Dreamhost please at least have the courtesy to write a new message instead of cutting and pasting the last message including the typo, “work together t0 bring things back”

Seriously??????????????

“If the service you received from us was exceptional, please consider tweeting your love for @dreamhost. It’ll warm our hearts, soothe our souls, and get you good karma at some point down the road.”

Yes Ron, I believe you, you are NOT working for DH and you have 2 million visit per day. I believe whatever you say. Maybe I should thank to DH for the downtime and show my appreciation for the uptime provided so far. Maybe I should make a donation now to keep their morale up.
How about that? Let us take it easy guys, let us join Ron in his yacht, we all need to relax…

I can’t belive that they are cutting and pasting. This is so disrespectful.

6 hours 6 mins ago: Outage verified: We are actively looking into resolving it.

6 hours 33 mins ago: Outage first reported.

If you guys are worried about your files being lost, you need to make sure you are backing them to some place other than Dreamhost. We all know the problems Dreamhost has. You can backup your entire mysql database using a shell script and cronjob, then send the backup to Amazon S3 storage. Do the same thing with your files being synchronized to Amazon S3 storage. Amazon will charge most of you a few bucks a month for that service. If you have it running daily, you’ll be at the most 1 day behind. You can find scripts, etc. via Google.

Thank God its Sunday, at least I’m not looking too much traffic as this is always the lamest day… hope to be back online soon!

Q: What’s the difference betwen Ron and a cockroach?
A: The cockroach probably has a sex life.

6 hours 14 mins ago: Outage verified: We are actively looking into resolving it.

6 hours 31 mins ago: Outage first reported.

@Gp we are all waiting for good news. I hope everything will back as it was… We use VPS for activeCollab which is our project management an collaboration tool. I cant imagine tomorrow if this issue cant solve :(

Lets talk serious now ..

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

This kind of message says: APACHE is running, but the DreamHost Staff have locked it down as Maintenance …

Why they did it?

Remember the issue related to Database last week?

Maybe they have found a serious security window at their system and was forced out to shutdown everything.

This is the only thing I can think about now .. since there is no explanation for this kind of issue.

We use VPS, Dedicated .. why a dedicated is affected if it’s not shared? Make NO SENSE ..

My other VPS just went out…

maybe this incident is related with the security issue of the other day?

This is what I’m wondering too.

That and whether anyone else here is naked.

Does anyone know what servers are affected? My web sites are up fine but if I try to access FTP it just times out and I can’t connect. I’m on stampeders. Anyone know a workaround so I can upload images by FTP. I’m using Fetch.

Dreamhost needs to invest A LOT of money if it intends on keeping any customers.

It’s selling a sub-standard product and once people leave them – they’re gone forever.

They’re becoming a laughing stock. The Zune of hosting companies.

@Rafael: they are not telling the whole story. I run on VPS Unity Asset Server which is NOT based on Apaache, it’s a custom SCC server and that isn’t reachable either, so THIS IS NOT A WEB OUTAGE, THIS IS A SECURITY BREAKDOWN. Dreamhost systems have been brought down somehow… I guess if this has been done wuth a virus then bye bye data as well :/

How many servers DH host? Probally they are using a UNIQUE PEN DRIVE, to update one to one .. How long time we need to wait more !!!

In other news my AWS EC2 micro instance has been rock solid (and the cost is almost exactly what a min. VPS with DH costs). Of course you have to manage it yourself, but with tools like puppet/chef this isn’t too hard.

@Chalise – Don’t think it has gone unnoticed that you have gone in and corrected your cut and paste repeated Typo!

YOU DON’T HAVE TIME FOR THAT @CHALISE! THERE ARE MORE IMPORTANT TASKS A FOOT!

I am sorry Dreamhost. This was the last straw for me. I cannot possibly continue to put my clients through this up and down yo-yo’ing. I have already started the process to moving to 1&1. It’s ashame because I really do enjoy your company.

Jan 3: Network outage
Jan 13: Network outage
Jan 18: Major outage
Now this? What is going on with Dreamhost? They used to be reliable, but seem to go down on a weekly basis now.

I go down on a nightly basis. Or sometimes more. ;)

@Rafael

Thanks for the explanation. I agree with you. I am on VPS too, getting the same messages.

By the way 2 weeks ago, all of my sites was hacked. I had to restore them from my own backups. I tried to restore them from DH and all returned the same error, they said, “for some reason” they could not do it. No reason was given.

A week prior to this incident, DH sent me notice which forced to change all of my passwords.

Something is up for sure and they are hiding it.

what’s this!! all dedicated servers are down and all VPS !!! is this hacking attack again???

I am in the same position as sesame above:

“I keep having to up my VPS memory allocation and paying more and yet, getting more insufficient memory allocation messages by the week.”

You expect me to pay more as of late when I’ve been DOWN MORE as of late. My site has been down for at LEAST 8 hours. I know some users above think sites don’t make money but mine does. Not to mention, Dreamhost was highly recommended to me as a fair, reliable service. Lately, not so much.

@DFT Games

I have 10 VPSs at DH, and 9 of them are down.

From these 9 VPSs, 4 not run apache, it uses Lighttpd and are DOWN too ..

There is something wrong on the history reported to us ..

Not good .. felling bad now about that.

6 hours 24 mins ago: Outage verified: We are actively looking into resolving it.

6 hours 52 mins ago: Outage first reported.

I have already started the process to moving to 1&1

I’ve already been with pair.net, 1&1, HostGator and ten thousand individuals of indeterminate gender.

Dreamhost is far better than those.

If you’re looking for options, you will have to go up the scale, not down it…

My client’s site is down: http://guerrillafitness.net and we started trying to adjust VPS resources and reboot the server several times and nothing happened. Then I hopped over to the Support area in the control panel and THAT’s when I noticed the alert about VPS/Shared services having an outage.

This alert needs to be on every page of the control panel. There was no point in me running through the motions of rebooting my VPS and possibly adding more strain to the server. I can only imagine thousands of VPS customers rebooting, allocating more memory several times like I did. Granted, I should have checked this website first but I always assume it’s a localized VPS/memory issues since we started getting VPS reboot emails lately even though I have a DreamhostPS manager instance running in cron that adjusts RAM resources on the fly. Please correct this outage and move the outage alerts to all pages of the control panel, not just the Support page.

Also, why has our VPS been rebooted a lot in the last month or so when we made no major adjustments to the site?

@Rafael: exactly, they are not telling the whole story… they insist on the Apache and ftp problem which cannot explain why Cron is down, custom servers are3 down and so on… They really should be straight with us instead of telling fairy tales to us :(

After half a day: WEBMAIL still inoperative. Passwords are mysteriously replaced by old ones. So that’s feels highly comfortable and safe.

Yeah.. I’ve also been told to up my memory..

On a VPS that is now only hosting one site.. whereas before I was told to up my memory it was 2..

And that site is almost entirely dead.

When this is all over, I’m lowering my memory requirements back, or switching to their cheap-o non-VPS service.

Seems like us VPS customers just get raped price wise, and really get little to no advantage. When I was paying ~$150 / month for massive memory it was all but useless to me anyways.. editing my config to try to get more memory per request always failed, and DH doesn’t provide any support for that.

How Odd!

I have received their email saying I should up my memory a few times recently. When I asked support they said it must be Google robots scanning the site so I should take action against that.

The Zune of hosting companies.

That’s overkill there, fella. Nothing is that bad.

Second weekend in a row I can’t provide the Ice Hockey live-updates service to my customers. They’re not happy. Neither am I. Have we got an ETA for service yet?

Can anybody hear a buzzing sound? Oh yeah its @Ranmdom Emosexual. Have you been playing catch with @Ron by any chance?

The way i see it, every hosting servers i use has a right to make one MAJOR crush once a year. simply because i don’t believe there is such a thing as 100% up time. hack, i don’t believe is 99% or even 97% up time.
so far i had about 3 years without major crushes (though with too many little ones).

I’m going to let this ULTRA-MAJOR one slide as long as it will be solved in the next few hours and as long as it doesn’t happen again.

UPDATE AND SAYS THE TRUE DH …

This joke apache and ftp history is not the true !!!

Andy: Yes.. suddenly google robots are causing my site to have memory issues?

After YEARS of this not being a problem?

Makes no sense. DH is having problems.

I just lowered my memory on my MySQL. Sent DH support a request, told them to figure it out.

6 hours 33 mins ago: Outage verified: We are actively looking into resolving it.

7 hours 1 min ago: Outage first reported.

@Anon – I will do the same! I thought it was odd.

This is a joke …

DreamHost .. where you host this status page?

Dreamhoststatus.com => 96.126.109.82

IP: 96.126.109.82
Decimal: 1618898258
Hostname: li366-82.members.linode.com
ISP: Linode
Organization: Linode

I think DH needs to learn with linode about that .. how to offer a stable service ..

http://www.linode.com

I am skeptical of the explanations we’re receiving for these outages. Last major outage, my server was down at the same time as everyone else, but I was told that it was unrelated. DreamHost said that in my case it was a problem with the kernel on my VPS and that they had to upgrade it.

Today I’m experiencing the same problems as I did then… no problem access static resources (HTML and images) via HTTP, but apache can’t reach my MySQL databases to power the dynamic pages.

I started experiencing problems last night at around 9:00 pm Pacific Time. Filed two support tickets, but still haven’t received a response to either. My customers are irate, and they don’t care that it’s not something within my control. They want to know what *I* am doing about it.

Sorry DreamHost… but this is likely going to be the end of our relationship.

@Andy IME Yahoo is the real culprit – they slam you unless you put a notice in your robots.txt to change their ping time

http://drupal.org/node/14177

use the crawl-delay modifier…

@Jacob – Thanks for the advice, I will check it out.

In 12 years of having websites, I have never been down. Until DreamHost, that is. We Launched a major Internet marketing campaign at 6am this morning. Who knows how many people. Opulent find our website, costing us thousands of dollars or more. And, as you all know, this is a repeat act. DreamHost is down quite a lot compared to professional companies.

One thing is for certain. We are leaving immediately.

I just checked linode web site.

I liked it.

@Random Homosexual

We all know that you and Ron are working for DH. Please GTFOH.

In 12 years of having websites, I have never been down. Until DreamHost, that is. We launched a major Internet marketing campaign at 6am this morning. Who knows how many people cannot find our website, costing us thousands of dollars or more. And, as you all know, this is a repeat act. DreamHost is down quite a lot compared to professional companies.

One thing is for certain. We will be leaving DreamHost immediately.

@Jacob : How can yahoo be slamming my site… when it’s been down for 6 hours?

MySQL still reporting major memory usage…

And again..hasn’t yahoo ALWAYS slammed sites? This was a sudden change.. my site went from barely using any memory.. to suddenly using 100’s of MB.

Makes zero sense.. it’s a DH issue.. I’m standing by that considering the memory usage during a DOWN TIME of the VPS web.

Ok, now DH is moderating the messages :) That’s a huge step forward!

This only confirms our things about there is something more than a simple apache / ftp issues

Response I got:

First, I’d like to apologize for the use of a canned response to this support issue. We have identified and are actively working on a fix for the problems you are experiencing with FTP and Web services. This issue is affecting a large subset of our customers and as such our system administration, data center operations, and development teams are all working on resolving these issues as quickly as possible.

We are making this information available, along with updates, on the http://www.dreamhoststatus.com/ website. Please follow this url to keep up to date with any future updates.

http://www.dreamhoststatus.com/2012/01/29/web-ssh-and-ftp-services-for-a-subset-of-vps-shared-and-dedicated-machines-down/

Again, we do apologize for the lack of a personal response to this support request and if you have any further questions please let us know.

Thank you for your understanding in this matter,
JJ Galvez
DreamHost Technical Support

I guess that tweeting @DreamHost and @DreamHostCare and tag with #DreamHost becomes due now

@Ulas:

No, I’m not working for Dreamhost. And I’m an EMO-sexual, not HOMO-sexual (although I have played those before and I love role playing, especially if you’re a top).

However, I have “been around” the web industry and I hate whiners.

Does this outage suck? Yes.

Is it Dreamhost’s “fault”? Perhaps — I don’t know.

However, the behavior of the people leaving comments makes me sick. You are behaving like a bunch of monkeys. You threaten, cajole, insult, and whine. What you don’t do is make any realistic statements.

In fact, I’d wager that it’s people like you who make hosting as expensive as it is, because you are totally unrealistic and irrational.

264 comments… almost no content. For what? For you to express your discontent? And what positive good comes of that?

In the meantime, some of us would like to come here and read a real update on the problem, specifically whether or not it’s related to the hack. We’re also thinking of Dreamhost employees who are not getting to loaf on Sunday (or go to the religious service of their choice) while they struggle through this mess.

Am I just as concerned about my web sites as you are about yours? Yes. I’m just taking another way of expressing it that I believe is more rational.

Now, let me hear from the rest of you about how your behavior is:

* Realistic
* Practical
* Constructive
* Rational

I’m open to all statements (and just about all offers, especially if you’re waify or wispy or ideally, both). I don’t have AIDS.

A lot of time. Too much time

I’ll second the request for a list of *which* subset of servers is affected. I noticed my site was down and assumed it was related to this, so I didn’t investigate further. Then it turned out that static pages were still working, but database-driven pages were not — which means that it’s a completely separate issue, which I could have started troubleshooting on my own, or reported, much earlier.

Curiously the post states:

“Please note that this only affects web VPS/dedicated and shared web server FTP services, no other services or servers (MySQL or mail, for example) are affected.”

Yet any of my sites that rely on a DB are down.

Who here gets the sites back?

My sites keeps down and @twitter we got only down info messages from many users

Time to leave Dream Host…I´ve seen this many times before…A company that gets bigger than It can handle…And the problems that come with this will not be solved in one day or night…It´ll take time to get the services back on track.

And there you have it:

This was caused by several software packages that were botched and/or removed by our packaging system. This was a new, unexpected bug that is unrelated to any of the outages earlier in the week.

Unless we suspect DH of lying, we have our answer. I’m going to treat it as honest, since DH have never given me any reason to believe otherwise, and go on with my day. I suggest you all do the same.

Does anybody noticed as they said?

I suggest all customers DEMAND a refund for the downtime we’ve incurred this month. I cannot log in to my own site- i am paying 125 bucks a month for VPS and nothing but problems

Random Emosexual: Yes.. their explanation is essentially “Hi, we are huge amateurs.”

It’s always that way.. a configuration error? Yes DreamHost.. I expect you to know how to configure your servers.

Yet the DH defenders will bash anyone for this ridiculous expectation.

ALL my sites still down!

Over the past two years, DreamHost has rapidly transitioned from the Best to the Worst. By far. It’s absolutely outrageous; all my sites and client sites have been down for 7 hours, with no sign of an end in sight? And there were multi-hour outages a week or two ago also?! No courtesy emails, which I’m 100% sure is because they’re praying that most of their customers won’t notice what happened. Maybe this is acceptable for customers with mental problems, or dialup modems, who still don’t know the difference between “AOL” and “the Interwebs”… but it’s absolutely insane for anyone with an IQ higher than their belt size.

BTW, I see there are 226 comments before me… but did everyone notice the message saying “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.”???

I guess the retards at Dreamhost don’t have the “skillz” to disable comments on their comment section. Pathetic.

I’m disgusted.

We’re still down. Going on 7 hours now. Our VPS was down on Jan 18 for a few hours with similar symptoms: The memory graph shows a 2x spike and then pages hang forever on loading. Sometimes they load partially without css. First reaction is to up the usage rate, then reboot server both to no effect. FTP and shell access goes down intermittently at first and then for good. logs show no suspect slamming or odd activity.

My VPS remain down.

DH is lying, that’s for sure. My Asset Server IS NOT MANAGED BY ANY OF DH SYSTEM still it’s down. I do understand that it’s hard to tell us something like “Sorry, we’ve been hacked and don’t have a clue about how to restore the service”, but honestly, many of us are IT people, so please DH, stop telling impossible stories and come up with something a little more credible.

I presume this problem has not been fixed at the time of me writing this 18:09 gmt

@Solo – I am in England and my VPS is still down at 18:10 GMT

PS47237 stil down! Is there any VPS up and running?

I think is everything down .. 100% customers

Cheers Andy time to watch a film then and pray we dont lose to many customers.

@random homosexual

Whatever you do serves for one purpose : defending and rationalizing bad service. Cut the crap about being rational and realistic. Reality is you are paid to defend them just as Ron, the only difference is you are less retarded, making better rationalization, making it less overt. That is why you had to write a one full page of explanation about why you lick them. We are losing money, credibility every second and you cannot call this “whining”. If you were losing money with us here, you would not be writing a full A4 page as an attempt to rationalizing and intellectualizing all the FUBAR going on here. You and Ron are selling yourselves (Don’t know how) to “taper down” the tension here. Please get lost. You are done here, go and get paid before they fire you.

May be worth asking if we can be moved to the server that manages this comment page as it’s the only thing that stays up lol

PS57991 still down!

It’s now going to 12 hours outage ..

INCREDIBLE !!!

@Solo – LMAO!

All of my sites and all services are down for 8 hours now.

I manage to entirely separate VPS accounts.. one my own, one for a friend.

Both still down… as far as I know, accounts created years apart, totally different underlying servers.

But they always both go down in these major outages.. despite DH’s claim that the problem is isolated.

According to mon.itor.us we have now been down for over 8 hours!

@anon:

It’s always that way.. a configuration error? Yes DreamHost.. I expect you to know how to configure your servers.

“Configuration error” covers a lot of ground. Wrong packages installed, bad software, conflicting instructions, missing config file, etc. You know how big the Linux operating system is (and getting bigger) and then on top of that you add all the web hosting stuff. It’s huge and full of possible screwups. Not to mention that open source software teams aren’t the best at letting you know in advance when an update brings huge changes.

@Ulas:

Again, I’m an EMO-sexual not a HOMO-sexual.

Whatever you do serves for one purpose : defending and rationalizing bad service.

I’m refusing to jump to such conclusions. We don’t know yet what has happened. I think everyone here is making a big deal out of an unfortunate event.

Let me ask you this: of all the comments here, which provided any useful information or helped the situation? Or are you all just venting?

aww for f#ck’s sake.

This page and status. dreamhost is hosted by LINODE.

I am currently reviewing their page and very much impressed so far.

@Random NumptySexual – You have become very boring!

What can you people do “fast” to bring the website online? I’m loosing money here!!! How much more time it will take to be back on-line??? Please advise!

@Random Emosexual: I understand the complexity. That’s why you hire people who know what they are doing. It’s what the large company I’m currently contracting for does. Been on contract 5 years now, building multiple sites hosted across various servers with varying technology.

Have had no major down times on any of the various servers.

In that same 5 years? My VPS accounts at DreamHost regularly have down time, often extensive.

I also have accounts at other hosts.. I have some Amazon ECN, a GoDaddy account and DiscountASP.Net.

Outside of the big Amazon ECN fiasco last year, I’ve had no downtime with any of those accounts either.

So save your insulting attitude for someone else. Some of us are quite experienced and knowledgeable, and it’s my opinion that DreamHost has never been that great, and this past few months, has gone to absolute shit.

@random hEmosexual

Why are you doing what you are doing? What is your motive? Why would you choose to be the person in this crowd who gets yelled by everyone else? Why? Why won’t you shut up? Why are you so persistent and determined about your rationalization. Your behavior cannot be accounted for anything else. You are working for DH. Ron was down earlier, and now you are out of the game. Go and tell them that they need to send someone else.

I’m looking forward to moving all of my clients to a new host – can anybody recommend?

There are a lot of people complaining about Dreamhost service, just want people to know there is a forum where people talk about the best hosting providers at the best price, etc. Check it out, you may find what works for you: http://www.webhostingtalk.com/

I did :)

@Jim – Yes can we make a list of GOOD hosting companies?

@Ulas:

Why are you doing what you are doing? What is your motive? Why would you choose to be the person in this crowd who gets yelled by everyone else?

Because whiners have destroyed western civilization.

I don’t care if people yell at me or not. I’m going to take a stand for what’s rational.

The vast majority of people here are emoting without sense. Really, you think you’re going to threaten your webhost with moving your host? Horse puckey. If you’re going to move it, you’ll just go ahead and do it. This is just a bunch of monkey posturing here, and it makes all humans look like idiots that this behavior is tolerated.

Also, from experience, this kind of behavior is what causes hosting costs to go up. It’s the human problem as usual.

But don’t let me interrupt a good tantrum.

Finally, I can see the light at the end of the tunnel. OH NO! It’s an oncoming train!

@Random Whatever – Yawn!

We don’t know what caused the issues with the packages yet, and likely will not until service is restored everywhere and we can do a full autopsy.

DH, your webservers recognized they were hosting a load of whiners (see the comments above) and so they tried to commit suicide. I don’t blame them.

@Andy:

Just get naked and come over here. I can make you feel alive, either as a top or a bottom. Or both. Whips, chains? No problem. I’ll bring the whipped cream.

@random hemosexual

It is very convincing now when you say your motive to save the “western civilization”. :)

Yes, I believe that.

Do you realize that you are pushed to move your position from a “smart” rationalizer to an idiotic loser?

Seriously, we’re approaching down for 7 hours. WTF!

Hey guys, I was just checking the page before I got out of cell phone range, you guys seem to be missing one important FACT: Dan W is on it and we will be back up shortly. Everybody settle down…

Woo Hoo @Ron is almost out of Cell Phone range!

Why they not save an IMAGE before did changes?

If the changes fail, back the image configuration ..

what a f**k is that DH ????????????????????

Here now Ron comes to give a hand to FUBARed R.Homosexual.

Good job guys… We have not noticed anything, just go on doing what you are doing.

DH, you are really even more frustrating me with these folks…

it would only take me an hour to fully restore my server from scratch! come on.

All of my sites/services are still down, approaching 9 hours now.

I just got on after still having our sites down all day. Um, who is this ‘Ron’ person? Should I summarize? Here are today’s updates, minus all the comments saying “We’re working hard on this issue” and who they’re working with on to do it, since it doesn’t really matter to us. It just sounds like they don’t know what they’re really doing and are in over their heads…

Update Jan 29th, 10:20am PST: We don’t know what caused the issues with the packages yet. -dan w

Update Jan 29th, 9:50am PST: We are still working on resolving all issues with the affected servers. -dan w

Update Jan 29th, 9:25am PST: We are still working with our admin team now to resolve this issue. -dan w

Update Jan 29th, 9:00am PST: We have located the source of the issue. – Chalise

Update Jan 29th, 8:40am PST: We have located the source of the issue. – Chalise

Update Jan 29th, 8:25am PST: Our admin team is hard at work to get this resolved as quickly as possible. – Chalise

Update Jan 29th, 8:05am PST: Our admin team continues to work hard on fixing this. – Chalise

Update Jan 29th, 7:35am PST: Our admin team is continuing to roll these updates out to VPS host machines; this also seems to be affecting dedicated servers as well. – Chalise

Update Jan 29th, 7:10am PST: Our administration team is currently working to roll these updates out to VPS host machines. – JJ G

Update Jan 29th, 6:40am PST: A fix is rolling out for the FTP services on the affected shared servers as well. – JJ G

Update Jan 29th, 6:20am PST: Our admins have identified the problem. – JJ G

Update Jan 29th, 4:55am PST: No further information to pass on yet. – JJ G

Update Jan 29th, 4:15am PST: The issue with ftp services also seems to be affecting some shared servers. – JJ G

Update Jan 29th, 4:03am PST: A configuration problem on the VPS servers has been identified and it is being fixed across the board for all web VPS customers. – JJ G

My websites are down too.

is anyone’s service back up, as they claim?

you could at least let us edit our A records in our DNS until this is resolved so we could publish a ‘sorry we are down’ page!

Has a single customer indicated they are back up?

Because for an hour now they’ve been claiming to be “rolling out updates”…

Yet.. my 2 disparate VPS are both still not working..

And I see no-one reporting they are up.

This is all pretty typical too. Hell, nearly 7 hours ago they were claiming they found the configuration issue and that it’s “being fixed across the board.”

“We don’t know what caused the issues with the packages yet, and likely will not until service is restored everywhere and we can do a full autopsy. More info to come soon.”

I find the use of the term “autopsy” interesting. Will there even BE a Dreamhost tomorrow?

Totally off – down here !!

10 VPSs !!!

I think DH will call CSI or NCIS tomorrow to check the autopsy .. wtf DH this is not time to joke .. SOLVE THE PROBLEM !!!

Can anyone say that they are back up? Is there light at the end of this tunnel? Latest update makes it seem like some services are back but they are still working on others. Is that true?

No, my sites and webmail are all still down. This is a mess.

all still down

Down Too !!!

All still down

I’m still down. No change. So glad I changed to a VPS.

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Update Jan 29th, 10:45am PST: Not much to report since the last update. Our team of admins continues to work to bring up downed services. We’ll have more information shortly. -dan w

Shortly? Now its 10 hours +- … where it is shortly?

Checking servers for copyrighted content now that ACTA is in effect are we?

At least three of my sites are back up, and able to ftp to the one I needed to. Fingers crossed.

aaaaaaaaaaaaaaaaaaand still no FTP abilities!

Nothing has changed for the past 9 hours now.
Nothing is working, no light.

Just got my first DS with DreamHost. Good to know that my dedicated server wasn’t affected… Wait. It is. Oh well. Sit and wait and pray that my churchgoing customers don’t start calling.

Dreamhost is trollin us.

Yeah, all 4 of my web properties are down, down, down. A couple of times they have loaded — but it took like a minute for them to load. Luckily, all of my websites are my own. I SERIOUSLY fee for those of you who have your CLIENTS hosted here. Dreamhost is risking the livelihoods of many people. It’s not that I think that they don’t care — of course they want to get this thing up and running. But I’m sorry to say that, in business, you don’t get points for effort. It’s about delivering results.

I’m in a tough spot. I am having problems with php, but not .html. Because the Dreamhost posts give no specific symptoms or issues besides ftp and apache, I don’t even know if I’m part of this issue. Furthermore, I get a canned response from support saying I am part of it. However, because its canned I can’t trust it.

Webmail still inaccessible here.

The Tracking Id of my issue is 4713544. People are dying, please help!

I just want to know what is going on. Please tell us what is really going on ! It is so clear that this is something serious.

AGGGGGG!!!!
You guys are the Titanic of hosting companies!!! I am bailing out!!!

@Ulas C – here is what is going on

Update Jan 29th, 10:45am PST: Not much to report since the last update. Our team of admins continues to work to bring up downed services. We’ll have more information shortly. -dan w

RELAX

@Mikey

Totally agree. I am damn sure that they are doing everything they can. But this is not a free service.

but… your accounting office keeps charging right on time!

7 hours 51 mins ago: Outage verified: We are actively looking into resolving it.

8 hours 18 mins ago: Outage first reported.

@ron

Thanks. Much relaxed now, since it is you who is saying it, feels like DH customer support.

I’m speechless…

@Ron, BTW, do you guys take turns, I mean, you and Random Bisexual Person, looks like you guys start BS service uninterruptedly but one at a time, huh?

I have one client who has had to suffer these down times more than any of my other clients, it is beyond painful to keep seeing these issues on an almost weekly basis. What I’d be interested in knowing is whether DH is hosting companies that are provoking DDOS which is then affecting everyone else. Whatever the case I for one want to know what’s happening over there.

@lee
“Whatever the case I for one want to know what’s happening over there.”

Feel the same way… Come on DH !!!!!!

you say mail is not affected but I can not get into my webmail service. Is this still an issue?

Do you have any idea about the damages you are responsable for with all this services disrupots, guys? For me is enough!

you guys are fantastic. one more failure like this and you will lose one more customer.

it’s 6 freaking hours since it all went down. my shop isn’t working, i’m not making any money.

that’s not the reason i moved my sites to you from my previous hosting provider.

c’mon, make it work finally.

All of my sites that are just using web services (i.e. static HTML, JS, and CSS) are up just fine. However all of my wordpress installs are still down. The only conclusion is that MYSQL services are in fact down. PHP info still runs just fine. What’s going on? Don’t tell us that MYSQL isn’t affected when it clearly is.

It’s a very very big #fail for your company. My webservice followgram (50k user register) it’s down in your VPS.
I search another sevice.

8 hours 7 mins ago: Outage verified: We are actively looking into resolving it.

8 hours 24 mins ago: Outage first reported.

Chat is currently offline.

My VPS is back.

ajm, what is your VPS number?

Can you enumerate all of the VPS were affected, and in your updates which VPS were brought online?

That’s it! Two outages in two months! We’re out of here! AWS EC2 NEVER goes down…

My VPS is up, I can ssh into it. When I get in there, the /dh/ server that my httpd launches from isn’t mounted. It being a vps, I don’t have access to an fstab or anything to mount it. It seems to be controlled from the host machine’s config file for my vps. I wonder if we could install nginx and at least start serving a frickin’ page.

What a shit hosting company.

I think webmail must be affected though I didn’t see it explicitly listed. I keep getting an “Unknown user or password incorrect” error and I can confirm that both entered were correct as I am still able to check mail via IMAP4 on my smartphone.

ftp accounts does not work a full day, when the problem will be solved?

Who are UP again?

We still down

I too have tried another service. I’m trying bluehost now because of ALL the CONSTANT problems with dreamhost and the total lack of good customer service.

Reliable web hosting services are out there. Most have live support. DreamHost ranks at 62%. That is the problem.

http://www.webhostingjury.com/

So, funny story. According to my panel, my DS is up. ;) But I can’t ping it or contact it in any way.

My VPS is 48771, still everything down.

Update Jan 29th, 11:25am PST: We are making good progress on getting the affected servers back up and running.

I can’t see any progress .. all is down for me ..

Me too…everything is still down.

My VPS (ps17228) is still down. Must be a pretty GD big “subset”.

Dreamhost – great for hobby/non-mission-critical sites, shit for anything that requires a website to be UP. C’mon guys, I mean really? Is it that difficult to keep all your services up? When things break at DH central, they always break in a BIG WAY with hours, even close to a days of downtime sometimes.

My VPS is still down, ps52131

Thanks for the info and updates – I know you’re working on the problem, so (unlike most commenters here) I’m not going to bitch about it.

Current status of my VPS: static HTML sites are showing up OK, PHP sites are still down with either 500 or 503 errors.

BTW, for those ranting in the comments here: I’ve been a web designer for 15 years, and hosted sites with a large variety of different web hosts. In my experience, downtime episodes can and do happen with all of them. No exceptions. The main difference between hosts is in how they handle it when it does, and I’ve generally found Dreamhost to be a lot more responsive and more transparent than most. Most web hosts don’t have any equivalent of this blog, and they certainly wouldn’t let all the negative comments stay up.

During my first few years of doing web work, I used to completely freak every time a site was down too. But after a while, you realize that things like this are going to happen every now and then no matter where you host. All you can do is just take deep breaths, learn what you can about what’s going on, communicate with your clients/friends/anyone else you may host on your account, and realize that this too shall pass.

No, Dreamhost isn’t perfect, but neither is anyone else, and in my experience they’re actually better than most. If you’re going to freak out and run off to a new host every time you have any downtime, you’re going to be changing hosts on a very regular basis.

AW come on DH I know I owe you $1.35 but I will pay …..honest lol…………..smile its easier that frowning

Tired of this (long) outages too but never found a good alternative that gives you total access like DH. Any suggestion?

Spidersilk is taking over from Ron and randomhomosexual now. Their shift ended.

Spidersilk is also coming to us with “years of expertise” and trying to calm things down for lamers here.

still down!!!!!!!!! Thats a joke DH? All day long sites/VPS is down…..

PHP pages are not working .. only HTML pages :S

Still down. I hope they can solve problem today.

You are late guys.I have just hired servint.I hope wont have any server issues

Me too…everything is still down. :( I hate this down.

Everything is down, approaching to 10 hours now.

We have now been down 8h+ (we are an European company, meaning that this is most of our business day)
After the incident last weekend this is simply too much :-(
Can anyone recommended a more reliable company?

i’m going to advertise everywhere to people not to get anywhere near your poor service … you lost alot of customers because of your weak performance and communication

Is the biggest down of my life.

I have to agree with spidersilk. Although I have only 5 years under my belt, and 3 with DreamHost as an independent, I have been on development teams that created sites for rackspace, GoDaddy, and several other large networks where I have experienced large scale outages like this. Once, a site we did hosted on RackSpace went down because someone physically unplugged a UPS that backed up another UPS, that backed up a battery box, that backed up the dedicated server. The entire data center died a slow anguished power hungry death because a fail-safe device many layers back from the initial server caused an entire data center to crash. It took them hours to recover, because they literally did not know that there was a faulty power outlet that allowed the natural force of gravity and a lack of friction to cause the VPS to become detached from the wall. It was crazy, but allowed for some community building downtime with my other local web geek buddies.

All in all, I’ve seen them all fall. We’ll be back up soon hopefully. I am optimistic.

the time has come the walrus said.

“Update Jan 29th, 11:25am PST: We are making good progress on getting the affected servers back up and running. A good portion of the services that were having problems are now back up and running. Our admin team is working at full force to restore service to all the remaining services that are having issues. The team that was handing restoring shared hosting services is close to finishing apache (http) and FTP services updates/restarts. Our dedicated/VPS team are still working out a couple of kinks on the backend and will then be sweeping through and fixing any remaining down services there. More info to come soon. -dan w”

Interestingly no one here reports a “back up and running” yet, wonder where that “good portion” is.

I’m still down. Is ANYONE up yet?

I can see few FTP accounts from our VPS’s are back to operation, but websites still showing 500 and 503 errors for PHP files.

HTML files are working fine for these sites that I m able to connect FTP now.

I can not at this time report any positive change for my DS. I am DS2710 and I am still down.

@Lee welcome to the team… (Ron, Random Homosexual, Slidersilk). Hope you have a better shift. Theirs were terrible…Good luck.

just come up after a VPS reboot fingers crossed

48771, still everything down.

I have moved my sites to vps two weeks ago and they are all down 2 or 3 days totally.What providers are worse than this performance ?If you know worse than Dh , I will move there asap!

Ulas C. I am taking off early. I just thought I would share my story of other downtime. Trust me, I am not very happy right now either. Just thought I would try to ease the pain. No need to insult anyone here. We all want the same thing.

Ok… downtime happens… I’ll get over it.

But guys… the least you could do is provide coffee and pastry on a Sunday morning when we have nothing better to do but twirl our thumbs waiting to get to our websites. ::smirks::

Is it fixed yet? LOL

i admire your hard work.
but don’t forget to have a reset and a cup of coffee.

Anybody fearing how a prolonged outage like this might affect SEO/SERP? I can’t imagine that Google is too happy about giving high visibility to websites that are offline for a whole day. I sure hope that Mr. Dreamhost dropped a lie to someone at Google to not hold it against us [not that Google would necessarily care.]

@Lee

Did not mean to insult anybody, but I don’t think I agree with your comment “we all want the same thing”

Go ahead and check some comments made above. It is so irritating when someone works for DH and not revealing his/her identity. İt was so obvious for some guys here, maybe not as much for you.

Wow. This is starting to get really out of hand. It’s sad, but this is exactly why I don’t recommend Dreamhost anymore. I’m still hoping this bout with downtime will go away, and stay away for another year or two.

Still not mention about web mail being down? Sent in my support ticket early this morning and no response. I am guessing the current VPS outage has all hands on deck. Lets which one of my customers gets pissed off first. I moved to D for this very kind of sketchy service.

Dreamhost now claiming to be rolling out updates for the last 2 hours…

Still not a single user reporting their sites are back up.

My 2 disparate VPS, both still not functioning.

They are now claiming the majority should be up.. where are these people DH?

I have 3 domains on my dedicated server 1 domain is down and 2 others works fine!!!!!! strange

Problems like these don’t just happen, there has to be a good reason.

Also maybe you need to up your staff amount, there seems to be quite a bit of us here waiting.

48771 globally down for 10+ hours as of now.

I’m guessing that dedicated is priority. My shared has been down from the very beginning and continues to be, and I predict it will be for quite a while…

8 hours 48 mins ago: Outage verified: We are actively looking into resolving it.

9 hours 15 mins ago: Outage first reported.

Ulas C,

I just meant that we’d like to see some progress. That’s what we all want. I am not a DH employee. I just want my sites up, but I see your point. Seriously though, good luck to everyone. I am gonna go have a beer and some pizza and turn my phone off.

Lee

I’m really happy with the progress so far. Soon we will all be back online!

January 29th, 2012 at 12:03 pmIncreasingly Frustrated Says:

I’ve stopped reading all of the complaints on this page and am now just using it as a place to find names of other hosting companies!

And FYI, has anyone else noticed that so many of us are complaining about all of the robot-emails telling us that our servers had to be restarted because of memory issues? I’m SERIOUSLY beginning to wonder if that isn’t some purposeful action by Dreamhost to increase revenues. In my particular case, a VPS with the same 5 WordPress blogs which have all had traffic DECREASES, are suddenly maxing out memory? I don’t think so. I think I’m being scammed into spending more money.

My worst downtimes were 1) at a small company in Switzerland and 2) at another large American provider.

In Switzerland, the boss had the habit of paying its network bills the very last day, at the very last drop of patience of his provider (the larger Swiss Telco). Once, he just missed it and went abroad on a long week-end. So the telco switched him off the very next day. And when he noticed, and tried to make the payment, they made him (and us) wait, and wait, and wait. I quitted him.

And at the American provider, I happened to have my websites on the same disk array which contained data that the FBI wanted to bring down. The federals just came down one morning, went to the server, shut it down the hard way and took all disks, including all backups. And as far as I know, they never gave them back. I quitted, too.

So DH, well, that’s not that bad.

My clients’ home page has been up all the time, but the inner site which is through wordpress is still down. Also emails have been inaccessible all morning too. There is definitely more to this story. We had been under the impression that a personal VPS would prevent this kind of thing from going on.

All – my VPS (ps73253) has been down this whole time, and continues to be down. So I don’t know how much of a priority we are compared to everyone else…

Increasingly Frustrated: Do as I do.. I lowered my memory back down, and sent DH a request this morning to figure out what their problem is.

I have the same issue.. I actually went from 1 site to 2.. and the site that is up?

It’s almost entirely a private site.. it’s a VBulletin forum where only people registered and added to a special usergroup have access to any real content.

And it’s been up for 6-7 years now.. used to get a lot more traffic, and was open to the public. But suddenly in December it’s memory usage jumped?

And this is happening to a lot of DH customers, all at once?

Yeah, really does seem fishy… either DH is ignoring a problem on their servers, which should have been evident.. or they are doing it on purpose.

A MySQL backend for a VBulletin forum that has maybe on average 20 posts per day shouldn’t be using 400mb of RAM.

Why are Shared Server customers being worked on before VPS and Dedicated servers?

Ridiculous!

@Increasingly Frustrated – I’ve noticed similar. I’ve got a small wiki and small forum, and it seems randomly memory usage just creeps up and up and up. More items get added to both, sure, but that should not increase memory usage since it is all just stored in files/sql and accessed as needed. Number of users has remained steady.

Our dedicated and VPS admin teams have been making progress on getting services on those servers restored as well, but we don’t have any hard stats on how much more work needs to be done on those yet.

VERY GOOD .. CONGRATULLATIONS !!!

THIS IS THE EXPECTED SUPPORT TO WHO PAY HUNDRED OF DOLLARS TO HAVE A VPS WITH YOUR COMPANY !!!

I’ve gotten the same goofy msg about memory use and I KNOW there has been no jump in traffic to my little hobby sites. It’s extremely fishy.

r e l a j a t e

I started to migrate last week and had to pause for other priorities…
DAMN, I regret it…

30 domains down for a whole Sunday!
…just waiting for my FTP back to get the hell out…

I am moving to 24shells (price at first, very nice support for starting up).
(don’t go to TMZ, terrible support, I asked for refund in the 1st week)

In my research, I had these other companies with very good references (specially KnownHost) in WebHostingTalk
KnownHost
FutureHosting
Linode
ThePrimeHost

Why are shared servers higher priority?

Ron Says:
I’m really happy with the progress so far. Soon we will all be back online!

Haha :)

What progress? This guy insists on playing the idiot here.

I have got an e-mail from support:

There has been an update on the service interruption
you have asked to be notified about:

Outage resolved: We believe the issue to be resolved.
(2012-01-29 12:07:47 PDT, 0 secs ago)

BUT STILL ITS DOWN!!!

FTP is back on server vela. Finally.. :)

I have been down for many hours now. I am new on Dreamhost but already I am disappointed after just one week here.

I have VPS, but would like to shift to shared server immediately
I hope someone can help me because I have asked for DH support to help but nothing has been done to change me from VPS to shared server for the last 4 days.

I am sad to continue disappointing my readers.

And I am amongst those 5 that are not working?! Great…

down, down, down, everthing down on my VPS

for 10 hours 15 minutes

In the years that I have been with Dream Host, I have watched google, facebook, myspace, and others go down. Heck, Department of Justice and the FBI went down just last week.

During those years, I can think of only one issue that they could not resolve for me. That issue was way, way, way out of the box and it was something their service just wasn’t intended to do. The short is that I was trying to mow my lawn with a sprinkler & Dream Host is not my personal dev. team.

So yes, this outage is annoying as hell. But it happens everywhere & the rewards for putting up with what happens everywhere are much better with Dream Host than I have found elsewhere.

Thank you Dream Host for providing a valuable service.

Grumpy (@the moment) DH Customer

P.S. – I later redesigned my sprinkler and am about to mow the lawn with the thing. How? Because the support team explained the limitations in a way I could understand and I created a work around for my specific needs.

@Erkan

I received the same reply almost 7 hours ago.

It is a joke.

My website is not working for almost 20 hours, i want a solution

9 hours 10 mins ago: Outage verified: We are actively looking into resolving it.

9 hours 28 mins ago: Outage first reported.

One of my VPS servers came up about 20 minutes ago. A sudden rush in traffic nearly killed it again. As for traffic, I have a robot.txt file block every spider that comes to my site and I average around 700,000-1,000,000 page hits a day. Could be Google to blame for error messages and memory limits. My site handles around 75,000,000 “successful request” a month easy on 700MB of memory.

This problem is costing my site a lot of traffic and thus ad revenue. Being down this long is kind of insane. I understand a couple hours, but this is something else…

@Ulas C
Aslında arkadaşlar ile yakinen tanışmak ve ilişkilerimi ileri boyutlara taşımak istiyorum :) Bunca yıl hiç sorun yaşatmayan firma bugün evlere şenlik kumpanya düzenledi. Şaştım kaldım inan :) Lakin bol sinkaf eşliğinde kulaklarını çınlatmaktayım. İmzaları da pek manidar: ‘UnHappy DreamHost Notification Team’

DOWN DOWN DOWN

There has been an update on the service interruption
you have asked to be notified about:

Outage resolved: We believe the issue to be resolved.

*THEY believe :)

@Erkan

Derhal değiştireceğim, ilk işim bu hafta. Geçmiş olsun cümlemize.

after 15 days of disruption on site and the mail will not work again. You’re hurting my client and my company. I need answers

Our FTP is working again at ALL VPSs from our accounts.

We are still experience issues at HTTP .. pages can´t be loaded.

Just wanted to pop in and say my sites are back up and I am a VPS client. People were asking if anyone was up, and I am. I have been down since the beginning.

FTP on vela WAS back, but, down again. Was a good 30 minutes.

looks like it might be time to jump ship , had a good run with dreamhost but of late it is a bit dodgy , vps was up and down more times then a 2 dollar hooker , so upgraded to a dedicated and still got problems

No FTP here. Just 503 service errors on wordpress/vbulletin pages. Simple html pages have loaded fine all day.

All down, for 10 hours and 30 minutes.

Enough is enough !

It’s convenient that they take their live support chat down whenever something like this happens.

My websites are not up yet >:E

It’s been over 9 hours already… half-day’s profit lost :(

@Ulas C paylaşımlı sunucuları VPS’den çok daha sorunsuz. Yoğun bir trafik yoksa tavsiye ederim.
Önemsiz siteler için kullanılabilir. Bunun dışında önemli gördüklerim zaten VPS üzerindeydi fakat durum ortada. Sunucuya ulaşır ulaşmaz ne varsa ben de taşıyorum.

@Mikey I have about 1/100th of your traffic and I’ve had it complain at 400MB.

Google will probably not be too happy as well… :(

Hello,

This is just a notification that your message has been moved to our severe problem queue. This queue is for major outages that cause a large amount of support to come in. Our goal is to resolve the problem based on the information you have provided. We will then send you an update once the issue is resolved. Depending on the problem it may take longer than 24 hours for a response. Please hang in there, when messages are moved here your problem is given top priority. Please note that callback support is not available for messages moved here.

Thanks!
The Dreamhost Ticket Moving Robot!

We have now been down 9h – this is my business and i lost money and clients today – this is unacceptable!

As a freelance web developer, the one day that I can get things done on is of course offline. I’ve been waiting the entire week for this weekend, and I’m sitting here doing nothing. This is intolerable!

I have about personal sites, client sites, portfolio sites all hosting here, all down for 10+ hours. Hopefully when this is all over we’ll get back half a day’s hosting fee – at $100/(31 x .5), that’s like $6! That will more than make up for lost sales, lost ad revenue, angry clients, and pissed off search engine bots.

I’m starting to get displeased.

down again???? wth?

ps63325 still down, any updates?

This is unacceptable, this is unacceptable !

Give my money back !

@erkan
Server reboot yaptım , hepten gitti, geri gelmiyor…Benim siteler çok yoğun 2 tane forum sitesi işletiyorum. VPS’e mecbur kaldım.

Our site is back up. I had logged a ticket about 3 hours ago and didn’t see anything, and opened another with A LOT MORE ALL-CAPS and quickly got the same notice as Rafael above. If your VPS site is still down you might file a new ticket–perhaps that’s how they’re learning who is still down?

still down!

ps57991 still down, come on!

I have never, EVER commented like this before, but I have to express how pissed I am that – despite paying almost $100pm for my VPS – somehow shared accounts are getting fixed while we’re still floundering.

I understand these things happen. I am certain that you’re doing all you can to put it right. But why are shared accounts being corrected before those of paying $1000s a month for VPS/dedicated services? What happened to my “priority” service?

I expect this to be addressed ASAP and an explanation why this has been allowed to happen.

FTP down again.

My sites are running. FTP on karpis was down. It now connects but cannot find my user directory. It seems to take me to what looks like a server root. Still, it’s progress. Thanks DH for working on this all night/morning.

@Ulas:

It is very convincing now when you say your motive to save the “western civilization”.

Where did I say that? Please quote.

My site is down still! I don’t see anyone saying their website is up. 100% sites are still down! They better give back our money!!

NOTHING UP and RUNNING! Just need to reach FTP and MySQL to move my sites.

Update Jan 29th, 12:25pm PST: Http services on shared hosting services should all be back up and running at the moment. FTP services are still starting on shared servers and is more than halfway done, so FTP service should be restored everywhere shortly. Our admin teams continue to go through dedicated and VPS servers that are still reporting down. We’ll update this post again shortly. -dan w

Are you kidding me? Why would you be giving priority to Shared Servers. I am paying 5 times as much as any shared server. You cost me big time credibility and reliability, you are done!

Give my data back at least. I am moving…Give my data back ! FTP is not working.

I’M ON STAMPEDERS, FTP IS STILL NOT WORKING! IN THE UK SO LOST A FULL DAY OF BUSINESS THANKS TO THIS!

My site is now back up, with a few intermittent failures. But at least its up.

Now down for more than 11 hours.
Sites do now respond to ping / traceroute but that’s about it – no web / no site back

We also just received out bi-annual bill and I guess like many others we are beyond furious because of the lack of any useful information provided by DH.

Why do shared servers get priority over VPS and dedicated servers?

Why do shared servers get priority over VPS and dedicated servers?

I rebooted my VPS server 1 hour ago and still not coming back. Any comments?

ok dedi back up , but ftp still down

Why do shared servers get priority over VPS and dedicated servers?

Still down here, I am paying $99/mo for a DS that I can’t even ping. I have been fortunate not to receive any calls from clients, but I can tell you this. I am not sure what services they restored, but it sure has hell aint mine.

The best part is that I pay the extra ~$110 a year for their phone support and when I added a ticket, it sent me the following automated email:

“This is just a notification that your message has been moved to our severe problem queue. This queue is for major outages that cause a large amount of support to come in…Please note that callback support is not available for messages moved here.”

So basically, I pay for support and to avoid giving that support, they “label” the issue “severe” and don’t provide support for severe issue. The ONLY reason I pay extra for phone support is for severe issues. I don’t need support for any other issues as I can handle just about anything else myself. I literally pay just for this reason. So I guess I will be requesting my money back or putting in a claim with my CC company.

For those who are suggesting that people who are on Dreamhost obviously don’t have mult-imillion dollar businesses I will say this. I personally do not host a multi-million dollar business with DH but I have at least two clients who combine for damn close to that. Their sites haven’t been affected but they both have page ranks of six with Google and both sell things. DH has been pretty good for them for almost 7 years but I shutter to think what would happen if their sites were down. There are plenty of highly successful businesses on DH. Both of my clients started out as nothing and only became big while using DH. Since everything was running smoothly they had no reason to move. I also have clients who make a decent living via Web sites hosted with DH. They aren’t millionaires but they are paying the bills and then some. One has her site down right now and is freaking out. If she loses a day it can mean she has a hard time making payroll and paying other bills. This is serious business. The idea that using DH means you can’t be making real money is absurd.

I personally host a four-letter .com with DH that is currently down right now. I cater to the financial industry so being down Sunday isn’t world-ending but it means that any updates I’ve been working on all week will have to wait another week to be rolled out. I need Sundays as a test day to make sure everything works as it should. If this continues much longer then I will start losing real money.

Our admin teams continue to go through dedicated and VPS servers that are still reporting down. We’ll update this post again shortly.

Are there more of these than shared servers? My guess is that most “serious” hosters went to VPS servers.

@Tom Kincaid
I was wondering the same… I’m sure VPS and dedicated customers have more critical websites than shared users.

Why do shared servers get priority over VPS and dedicated servers?

All sites hosted on my VPS are down. Nothing is working, not HTTP nor FTP. Real and reliable information to your customers will be appreciated!!

So what’s the root cause of the down time? Are you guys Hacked again???

@gustavo: Ditto. I got nothing.

Ulas C. asked “Why do shared servers get priority over VPS and dedicated servers?”

It is a “Numbers Game”. Each VPS and dedicated server serves just one customer yet they must be configured/fixed one at a time. Shared servers serve many hundreds of customers each, and one config fix gets many hundreds of customers back on-line. Plus, there are many more VPS and dedicated servers than there are shared servers.

Plus, in this particular outage, it would appear that “the problem is harder to fix” for VPS and dedicated servers, based on the Updates provided by DreamHost.

I don’t work for DreamHost, I’m just a customer. And, yes, I am on a VPS that is still down (OK, FTP and SSH works, but Apache is still down).

Homosexual, really can’t tolerate you now. It has been 5 hours since you were spreading BS all around. Any rationalization and intellectualization for the past 5 hours? Welcome to your new shift now.

A “serious” DH hoster.

“Why do shared servers get priority over VPS and dedicated servers?”

Good question. Shouldnt be the opposite?

still 100% down on ps5493

Still down. I can appreciate that no ISP can provide 100% uptime, but I must say Dreamhost is experiencing a lot of downtime lately and the length of those is getting worse… Otherwise Dreamhost is great. I’m sure most user would pay that little bit extra to reduce those downtimes.

My site that I posted as back online is on a VPS.

@ Jon Perkins

Thanks for the explanation.

The highest paid the last attended… Why VPS are still down?

They’re not back up, sites are still down.

We pay a VPS to get a better service than a shared server. We should get priority.

I am appalled at the lack of attention given to us that have DS accounts. The one reason I bought a DS account was to increase the availibility of connections as well as decrease the shared blunders of other users on my server. We did this to begin hosting high profile political campaign websites which generate lots of campaign contributions after hours and on the weekends. For the site to be down right now is potentially costing political candidates of mine several opportunites to receive donations through their sites. One of our sites is a US Congress race and has generated a substantial amount of income considering it is a brand new site. Please DreamHost, get us up and running.

I’m on a VPS by the way and all access is down, no ftp, no webmail either, my clients are furious:(

@Jon Perkins
In which way my site is lower than shared user’s website? Don’t you think VPS and dedicated customers have more critical websites than shared users that’s why they pay more? I don’t care about how many numbers they want to resolve but they should give priority to us than shared customers.

If you pay more you get better service isn’t that the real world scenario?

@Ulas – you think you are making friends here but you are not. NO ONE cares about your vps being down except for you. we only care about our own. so don’t expect any sympathy from us. and by “US” I don’t mean dreamhost, but the community of dreamhost customers who are part of the outage.

i really do hope you move your data somewhere else so i dont have to listen to you whine when there is a blip in service.

11 hours still down ps48770 with all services, HTTP, FTP, mysql, shell.
Rebooted about 2 hours ago, completely dead now, not coming back.

I have my sites on VPS and am (finally) back up. So that was what, 11 hours or so?

@Lee I’m in the exact same situation. My nightmare is that today would be November 6…

If we are over a REAL VPS or DEDICATED, things like this will NEVER happens.

We have VPSs at vps.net and each vps is independent when we talk about software installations, not resources.

Here at DH, appear to be everything shared, apache, ftp, everything …

It means the VPS is not a real VPS, and DEDICATED is NOT DEDICATED …

At a VPS or DEDICATED, the OWNER should be able to define if he wanna update or not his system ..

Here we are locked to just say “OK” to everything that DH did inside our VPS or DEDICATED servers

I’m not jumping ship, just hoping and waiting for some sort of resolution. This thread is starting to look like /b/. Maybe we could all just start some sort of storytelling game while we wait. That might pass the time.

Aaron,

I feel your pain man, and yes, thank god it is not November 6.

Lee

@Ron — You don’t even have a VPS. You are paid-by-shift DH employee… I understand you are annoyed with this because it has become overt. It is a bad job you are doing, very bad job, and you are not even doing it well. Think about it, it is not worth it, really.

GIVE US AN UPDATE .. WTF!

So much for getting anything done today. I hope we will all be getting big credits on our accounts.

Need an update, really, going crazy here.

This page’s update says all servers back up and running. NOPE. 503 Service Temporarily Unavailable. Bad Sysadmin! No biscuit!

Rafael,

I totally agree with you about VPS.

@Ulas – you think you are so smart, I’m not a DH employee, but just another troll on the interwebs, getting high-strung people like you all wired up for the lulz

9 hours 59 mins ago: Outage verified: We are actively looking into resolving it.

10 hours 16 mins ago: Outage first reported.

This is insane. I have a ton of sites that are down and I’m getting emails and phone calls and I don’t know what to tell them. I’m now looking into finding another hosting provider, preferably one that I can depend on. If anyone has any suggestions, I’d appreciate it.

I’m seeing whole bunch of activity from “dhapache” user, is this normal?:

7859 dhapache 20 0 220m 12m 1560 S 0 1.2 0:00.00 apache2-ps2xxxx

@DH Customer: The nginx managed by Dreamhost is also down, because of the same reason: there’s no /dh/ folder mounted.

My stuff is still down.

All of the whining and bitching isn’t really helping anyone. It’s lame.

I really do think that Dreamhost should provide a credit to folks who have been down today. It’s the right thing to do.

Uhmmm….. the update says all the services should be now back up and running – none of mine are. All on VPS….

@Ron — Yes, I believe you. You are NOT a “real” DH employee. You are just someone who gets paid to do this. Think about it, we have been down for almost 12 hours and you and Random Hypersexual are the only ones not complaining about this, trying to rationalize every BS. What kind of a retarded persistently acts like this in such a crisis? I will tell you, option 1 : you are getting paid, option 2 : you are REALLY cognitively impaired. There won’t be 3rd option sorry.

ps64596 – PHP sites like desktography.com are not working, but the httml deos

@Ulas- you should try and be more positive.

The fact that the last update was almost an hour ago is one of the more annoying aspects to this.

Still down. You now owe me $100 pet your own promise, Dreamhost.

Sorry, user xpsvideochat_api is not allowed to execute ‘/etc/init.d/httpd2 restart’ as root on ps30384.dreamhostps.com.

@Ron – Thanks bud. I will try to…Do you want to compensate people’s financial losses cased by your company today? I would seriously think about this if I were them…

BTW, what was your site again which was down, prove us that your site is down. Do you want to do it?

Down for 11 hours 23 minutes as of now.
Everything is down. Nothing is working.

It looks like I might be up for now. Checking with the DreamApp for iPhone and getting positive results. Some aree slow to come up, but are slowly ddoing so.

What a F**K you are doing at our servers?

/bin/ls: cannot access /dh/apache2: No such file or directory

DH good only for static html sites and cheap virtual hosting. moving all my sites now on other server…

All my sites are still down, but it looks like others are getting back online… first good news in 10 hours:)

Down all day here. It’s a really good job that the guys running this farce of a service aren’t responsible for life support machines.

DreamHost is a misfiring 1981 VW Jetta in a world full of 2012 Bentleys.

@Mikey

I use to have almost the same issues you do with your VPS, until I moved from Apache to Nginx. My site has stayed up and running all throughout this little ordeal because I am not using Apache. My resource consumption is about half of what it was by using Nginx instead of Apache.

I am running a WordPress site with some extensive plugins, and provide hundreds of songs and videos. I get an average of 4500 unique hits a day with an average stay of about four minutes. So obviously, I am pulling some resources on my VPS. I have installed JJ’s Memory Manager on my server that will automatically adjust my server resources as needed, so I can get a bump up or down when my traffic requires it. It seems to be working just fine.

Unfortunately, like everyone else, my FTP is still down. My mail is handled by Google, so that isn’t a problem.

On a different note:

As I mentioned in a prior thread, I am completely amazed at many of the comments that appear here, and how they describe how they are losing thousands of dollars because they chose to rely on a shared web service to host their critical business website. (or something to that effect)

Seriously?

Yes, having your websites down is an irritation. I am irritated when the windshield washer light comes on my dashboard in my car too. We want things we pay for to work. (who am I kidding? — we want things we AREN’T paying for to work also) But we have to realize that we are dealing with electronics here, and electronics fail. It might seem that it is the same thing over and over again, but that isn’t the case. There is a lot of equipment between you and your server, and a hellova lot of variables to go wrong. The larger the datacenter, the more there is to go south at any given moment. It could be a server or it could be as simple as a $5 power strip that causes a cascading effect. Sh*t happens. A squirrel gets into a power vault and BLAM! There goes power. A generator fails to start because it picked THAT time to be truculent, even though it passed a PM load test the day before.

It would be great to have the redundancy of a “Cloud” at DreamHost. The problem is, most of you wouldn’t want to pay that kind of money — otherwise you wouldn’t be here in the first place.

I am always seeing the people here threatening to “jump ship” or this is the “final straw”. They start looking for that other magical host that never goes down. After 30 minutes of angry web surfing, they end up staying here.

Still down – VPS running but no Apache, etc.

1hr since the last update sucks.

Tried moving my domain from VPS to Shared, this did not work…it did not complete… How the HELL to they justify putting shared hosting before VPS customers.. must be that there is much much more shared customers effected.. but still.. when you pay more.. u expect not to be put last.. I am in Australia and 20 staff are about to turn up to work .. in about 30 minutes.. I have a CRM that manages all clients online that all my staff use.. so it looks like no work for our whole company today… going to cost us a fortune.. I am sure many of you are in same position.. I am in Australia so it hasn’t effected work hours yet… but Australian clients are really about to feel it..

Thoroughly disappointed with dream host today now 12 hours downtime

David
London

Still down. Waiting for good news…

Im back 100%. Static pages and mysql

Still down…when told all are back up

10 hours 19 mins ago: Outage verified: We are actively looking into resolving it.

10 hours 47 mins ago: Outage first reported.

I’m guessing they’ll loose a ton of clients after this downtime… Including me… It is totally unacceptable to have 12 hours consecutive downtime. This is total BS.

down, down for 11 hours 30 min.

@Lee

What is your VPS number if you don’t mind telling?

ps54457 still down as of 1:30pm PST

I love how DH says things are working, when they obviously aren’t. Who’s driving this clown car, anyway?

Same here, I’m in Aus and I’m bracing for the phone calls that are bound to start coming in 30 mins if all my sites are still down.

my site is still down, any updates for ps63325 ????

ps48770 down as of 4.30 pm EST

As stated before, I had an FTP connection to karpis.dreamhost.com, but it could not find my directory. Now the connection times out again. Hope this info helps in diagnostics. Thanks.

14 hours down….

15 years designing web sites. I’ve never seen a downtime this long. It’s shameful. Get it fixed DH!!!!

This is unreal. I dont understand how all the shared servers come up before the VPS and Dedicated. Arent we the priority?

PS47237 still down… I dont understand why they dont update status more then one hour? Hey DH team are you still there?

I am still down….. This is crazy!

My VPS site just came back up.

They go out for lunch time, company is closed at this time …

“Jeff Says:
15 years designing web sites. I’ve never seen a downtime this long. It’s shameful. Get it fixed DH!!!!”

Apparently some “years of experience” holders such as Spidersilk, Ron and Random Hemosexual for instance have considered this issue NOT “beyond normal limits”. I tend to believe them to stay calm and optimistic… (!) :)

VPS still down, I have started the moving process. My customers and my self are loosing. I don’t think we are even getting the true story behind this outage.

I’m afraid this is the end of dh for me.

Shared get up first? WTF?

Update Jan 29th, 1:40pm PST: The shared end of this outage should be resolved at this time. Both http and ftp services are up and running on all shared hosting servers. Unfortunately, for the dedicated and VPS servers there are more specialized setups on those servers and it’s taking more manual intervention from our admins to get things back up and running again. Our entire admin team is focused on the remaining servers that remain down and are working on getting this up ASAP. More info to come shortly. -dan w

Dreamhost apparently think that 1hr15mins between updates is giving us ‘more info shortly’ = pathetic communication

Expensive and slow for a VPS this is probably the last straw for me at Dreamhost.

How is this different than the previous update?

@Ulas:

Apparently some “years of experience” holders such as Spidersilk, Ron and Random Hemosexual for instance have considered this issue NOT “beyond normal limits”.

I’ve seen much worse. I don’t know how angry messages on a web page will make it go away faster.

“The shared end of this outage should be resolved at this time. Both http and ftp services are up and running on all shared hosting servers.”

I regret to inform you that my shared FTP access is still timing out. I will try again in 5 minutes.

January 29th, 2012 at 1:50 pmMr. Conspiracy Says:

Is this the work of the Illuminati?

I’m on a shared server and can’t access FTP. Stampeders.

I hoping more details about the cause of the outage will be coming.

@Darren

NOW you’ve done it. What the hell did you expect? In a normal situation it can take up to 24 hours to make a move from VPS back to shared. But this isn’t a normal situation — your VPS isn’t working properly, and any account changes are going to just stack up until the problem is fixed.

And really, you are using a web hosting service like DreamHost as your company’s critical backbone? You do realize that you get what you are paying for, don’t you?

@Jeff

Really? Fifteen years and you have NEVER seen downtime like this? Google was down for three days once. I have been on a few score of webhosts that have been down for up to a week at a time.

@Anthony

Why would you assume there was a priority consideration here? Sure, you are paying more money than a Shared account, but that money is for resources, NOT some magical status advantage.

@Ruben

Don’t let the door hit ya where the Lord split ya. Good luck finding that magical web host that NEVER has problems. I hear they come with your own personal unicorn that farts multi-colored sparkles.

FTP no longer times out times out, but I get this error:

“Could not download “myDomainNameHere.com”. File not found.”

What about US ! you cost us loose 6000 daily visitors .

Random Emosexual Says:
I’ve seen much worse. I don’t know how angry messages on a web page will make it go away faster.

1. Sounds like you’re a longtime Dreamhost customer.
2. I guarantee you 573 ‘angry messages’ is a lot more effective at making something happen than 3 messages that say, “I’ve seen much worse”.

@Random Homesexual, it won’t make it faster but without people like you it would have been much easier so far.You are still rationalizing this, it is unbelievable you are doing this. “You have seen much worse”, hmmm.. All right then…Since you have seen worse, this would be acceptable? This is the most primitive attempt possible to “find reasons” to explain things. You, as a customer, are not supposed to find and explain things for the company, unless you are working for them. You, and we, as customers, are supposed to complain as needed, ask questions, demand clear answers, and if not given, to ask our money back. That is what you are supposed to do. So stop this idiotic moderation of this blog.

Dow all day. Still down…

All of my VPS are down. This is costing me a lot and my Clients are sooo upset they want their money back because they are loosing their sales and orders…

Dreamhost be ready… this time I am going to request for a full refund for the last twelve months. Where the heck is your 99.4% uptime SLA?

My entire team is pissed. It is time to switch to Amazon Cloud.

I understand that posting angry messages is not going to speed things up, but it’s the only place where the paying users can vent their frustration. And while I’m not as angry as some of the other people that have posted here, 12 hours is way over what would be an “acceptable norm”, and so people are quite well within their right to complain.

This is our 10th year using dreamhost–I feel like they still have us on the original machine we signed up on eons ago and it finally died. Time to replace some hardware????

Nope…. still down, all sites:(

Down for 12 hours now.
PS 48770

What sucks is the lack answers and customer service. I probably have 75 or more active sites that are down (I’m assuming after checking 5 of them with no luck). I never even got an email telling me that the VPS was down. WTF? I had to find out from a customers email asking why their site was down.

I don’t understand the ranting and screaming. I’m sure DH is doing the best they can to resolve this. I’m not saying it’s acceptable to have had so many problems over the past six months but being rude and childish is just silly.

I took half my sites to HG to test the waters and found some problems there, too. What I haven’t found on HG is the extended downtimes and the contant WP hacking/security issues that I see on DH. HG is a much more secure host than DH. I did have a serious issue with slow page loads on HG. They seem to have fixed whatever was causing that.

It seems to me that as DH customers we should ask the new Prez to address the downtime and security issues at DH. We should ask for more than lip service. He should offer an action plan with defined changes in practices and equipment. This would be a constructive way to deal with problems like we are seeing today.

Well it’s the final straw. The upside to DH is the ease of use but sadly I’ve outgrown that. Uptime is PITIFUL. It was nice while it lasted. I’m temporarily moving to hostgator.

Ulas C i used to be ps31128, but now I’m DS2710.

Most of my sites are up.

Also what I find funny is shared is good but VPS is still down AGAIN. Always the case the cheaper accounts get quicker fix. What a joke. I might actually check out the shared accounts here since they seem to work fine.

Please fix, my VPS server is down. None of my sites work.

-Paul Solt

eyebeat where do you come up with past 6 months? I’ve been with DH for years.

@DJ Allyn.

The site I attempted to transfer from from VPS back to Shared was not a live site. it was just a test and this sort of transfer only usually takes 15 minutes. it was only 1 domain with very few files.. I received an email saying the foot folder at both locations could not be found so the transfer failed.

Regardless of hosting a CRM on VPS…. I am not claiming it is more important than any other users normal websites.. for me it obviously is.. but any business hosting a website.. when that site goes down their business is effected.. my main point was.. Good Morning Australia… time for everyone to wake up and see that their sites are down.. we usually are lucky that most outages are during the night for us.. not this time I guess.. Dreamhost as a whole is a great service.. I have been with for almost 11 years.. the service is very broad and caters for most hosting requirements.. the accounts team is great when u can get them on chat.. and support is great when u get them on chat…… but this outage is huge I have never seen one this big in all time been with DH… being 8:30am here.. I am only hoping for us it gets back online very soon..

My site is still not up i’m on ps24490, anyone else lucky enough to get your website up?

10 hours 52 mins ago: Outage verified: We are actively looking into resolving it.

11 hours 9 mins ago: Outage first reported.

FTP on vela is back, again. I have a feeling it’s back for good this time.

My sites are down.
Testing also has shown error in the apache configuration.
Any predictions on the standardization of services?

I’ve been with DH for the past 5 years and it’s only lately that I’ve started noticing big outages. This is the longest one I’ve had so far…. all my sites are still down….

12 hours of downtime, big time loss of credibility to our customers, big time money loss, and some special retards can’t still understand the screaming and ranting. What would have had to happen for those of you to understand the simple and clear reason why people are screaming here? This must be a joke. While this historical scandal happening, whoever tries to defend DH is clearly paid by them or related to them. It is a possibility game.

Interestingly enough, none of the “volunteer defenders of DH” so far revealed their “down” web sites, and none of their sites has gotten up that we know of. All of them are here and insisting on cover-up operation.

down, for 12 hours 10 minutes now
ps48770

Just got a reply :

Hello,

Please accept our apologies for the downtime you experienced in your hosting services. Details on this can be found under our status post: http://www.dreamhoststatus.com/2012/01/29/web-ssh-and-ftp-services-for-a-subset-of-vps-shared-and-dedicated-machines-down/
Basically, web, ssh and FTP services experienced a disruption in service starting last night. Our teams have been working at finding the cause and to fix the problems. Currently, we’re confident that all shared services are operational. However, we’re still working on the dedicated and VPS machines. We appreciated your patience during this process. Please check that above status post for frequent updates.

Please note, this message is a reply to an outage that affected many customers. If you’re specific case is not related, please reply to this message with the details, so we can look into it for you.

Thank you,
DreamHost.

I’ve been able to load my site for roughly an hour today.

I’m definitely switching hosts. Totally fed up with their bad service. I don’t care that it isn’t their fault and they’re doing the best they can. It’s their job work as hard as they can to get the system back up. I haven’t been able to make changes to my website all day, and I just can’t afford to waste this much time.

ps49102 still down as of 02:00 PST

My VPS is still down, ps52131.

And Ulas, I understand why people are upset, I know I’m frustrated. But I still think screaming and ranting is a bit unnecessary as it will not make your sites pop up any faster. Pull yourself together, man! lol :)

My VPS is still down, ps59939, as of 2:14pm PST. I expect a refund for today.

PS14049 still down

My wife passed away lately she had a drink problem.

aww she was an alchoholic then?

No she got hit by a beer truck.

Looks like there is no relationship between the number of the servers and ones being down.
I thought there was initially.

My blog ps7420 is still down as well. Funny thing is that the site was working but php was failing, then I restarted the VPS and afterwards everything failed, except mumble and ssh connections which works just fine still.

I’ve had several client sites down all day 7AM to 5PM and still down. Clients are threatening us; looks like we have to move them just to get them back up.

I guess if we want to stay with Dreamhost we’ll have to have all our sites on a backup hosting provider for these daylong outages.

My FTP on a Shared Server just went down again.

Server Name is: zuben-el-akrab brock

Down for about 12 hours and 30 minutes now.
pS48770

11 hours 10 mins ago: Outage verified: We are actively looking into resolving it.

11 hours 38 mins ago: Outage first reported.

All my VPS sites are still down…

FTP services on lansky (shared) are still down as-of 5:22PM Eastern 1/29/2012.

@db0 — It happened to me as well, used to be getting 503 errors, tried to reboot about 3 hours ago, not getting anything so far, died completely. No idea why.

ps59368 Still down.

@Greg I’ve been with DH for a long time and reliability started to lag about 3 years ago has been gradually getting worse. The last 1.5 years seem to have been really bad. I can’t back up this “feeling” with data but that’s how it seems to me. I’ve had two instances of WP hacks in the last 2 years on DH with none on HG.

Still, I prefer many things on DH to the way they are handled on HG. I don’t know how bad it will have to get for me to take everything to HG.

I’m suspicious that the recent security problem has led to what we’re seeing today. Sometimes, it seems as if DH is infested ticks and fleas like a stray dog.

To others with FTP issues, mine was a problem with my remote path. Before the update, “/foldername” worked. Now the slash confuses things. My remote path works now as “foldername” without the slash.

voodoo (shared) ftp still down as of 14:28 PST 01/29/12

sites up and running.. it seems

On stampeders, still can’t connect to FTP!

January 29th, 2012 at 2:32 pmUnhappy Dreamhosters Says:

CCCCOOOOMMMMMMEEEE OOOONNNN..!!!

It’s Monday in here..!! and my VPS still down..!!! Can you imagine that our clients will be complain today?

I tried moving my sites back and forth from VPS to shared, but no luck. Now the outage is just VPS, so I’m moving them back to shared. Anyone on shared still have symptoms of this outage? Would be nice to start a separate thread for those ranting, and use this one for people trying to help muddle through this.

C’mon DH, this is just unacceptable, site down for hours now.. will move away soon.. pity..

@Maria – At least you understand why we are upset, maybe because you are upset too :) I understand that you have a VPS which is down. Some people here (Random HomoSexual, Ron and Slider_soldier_tinkle_spider) have consistently questioned why we are upset, they demanded that we stop “whining”. Because they have seen worse, and this is just “normal” to them. These “philosophers” taught us some life lessons. They have made their phiolosophical debut here in DH blogs…Quite a reputation. Thanks to them.

BTW, still down, still down…:(

Mine is back up.

ps12853 still down.
*sigh*

Still down. We have eCommerce websites running on VPS which are losing money every minute. Our fault for using Dreamhost I suppose. We need downtime to be minimal. Moving to amazon next week.

Poor Ulas C needs to be up. Sorry man. Mine is up, but I am feeling bad for those of you who aren’t. Mine is a DS, so all i can say is maybe they corrected shared, then DS, then VPS. I have no clue. I have not tried FTP. I am just happy that sites are resolving.

voodoo (shared) FTP is up! As of 14:38 PST 1/29/12

January 29th, 2012 at 2:42 pmIncreasingly Frustrated Says:

Still no downtime for their own site: http://dreamhost.com/

So happy for them that they are able to continue to sign up new customers while their existing customers can’t do any business.

If you have any social media account (twitter, facebook, google+, etc), let’s start CAUSING change to happen: Post this page address for the world to read! I’ll bet that will get Dreamhost top brass engaged in making positive change to this continually degrading web hosting company!

ps18214 Still down.

I am getting :
Not Found
The requested URL /cgi-system/php5.cgi/index.php was not found on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Increasingly Frustrated Says:
Still no downtime for their own site: http://dreamhost.com/

Actually their main site is hosted on Linode Hosting company which you might want to review :

http://www.linode.com/

Loved Dreamhost for a while, but these fluctuating outages and weird ram shortages are having me look elsewhere.

haha now they decided to moderate my comments…How pathetic…

(Ulas C)

I can’t stress enough how bad your hosting is. You guys are down more than any other host I’ve ever used in my life. In fact, if I didn’t have so many sites hosted with you guys I would have transferred ages ago.

still down

Approaching 13 hours…
ps48770

VPS servers were hit the hardest by this issue. Due to the number of VPS servers, it is taking our admins longer to go through and apply the fixes to them. Progress is being made at a quick pace to restore service to the remaining servers.

This is kind of what I figured. Something tells me this is one of those gnarly tech issues that can’t be automated yet is grindingly simple, and thus we think it “should” be to save our brains from the tedium. Poor DH staff having to do this stuff on a Sunday night. Hope DH doesn’t drug test.

Near 13 hours without web, ftp, shell, everything… ps34024 looks really dead :_(

Bad Service, Bad Support, Bad Hosting…

WTF http://www.thickdame.com/ is defaced and hacked by some Turks? Dreamhost?

Shared server Sinistra down. All my sites offline. Yesterday I had the same issue. And the day before. And the day before that day. This is unbelievable!!!!!! Did you hire as CTO the same captain of that damned cruiser sunk in Italy or what????

@Random Emosexual Says:
Poor DH staff having to do this stuff on a Sunday night.

Haha, this sexual guy is hilarious ! Poor DH staff ! Poor them, all of their Sunday is wasted…Maybe we can do something about this huh? You are a joke man. How much are they paying to stay here all day and moderate us? Would you mind telling us your VPS number or the site which is down? The web site links to your name is a porn site. IS that your site? That appears to be okay. So why are you here?

January 29th, 2012 at 2:58 pmKirsten Shilakes Says:

So far my site on ps74281 is still missing it’s /dh folder among other issues

Wow, my site has been down for nearly 12 hours! That’s got to be some kind of record.

ps50184 is still down.

Do you guys get it now? I said http://www.thickdame.com/ is hacked and defaced which means someone hacked DH again.

All of my sites are dead for 13 hours as of this minute.

ps53369 down 14 hours

@adam gerts — Is that your site which is hacked?

UC: Not mine but i saw the owner on twitter saying her website redirects to a “French” website. she doesn’t know. Apparently it looks like it is hosted with DH and it’s been hacked. I hope I can’t see that “index”on my websites.

“VPS servers were hit the hardest by this issue.”

WHAT WAS THE ISSUE??? A sunflare? Gnomes? Magic? Isn’t part of the fun of having a VPS that you don’t just bite the dust along with everyone else when something goes wrong? Clearly now that I’m trying to do something other than a hobby, it’s time to find a serious host.

Also, it seems like other people were having the RAM issues, right? My (WordPress) site’s traffic never went past 600 page views/day, and I had to move it to a VPS because it was getting its scripts killed on the shared server for exceeding memory allotment, and on the VPS it was spiking past 400MB in the middle of the night, with no traffic. I killed all plugins, optimized WordPress by enabling the cache and whatnot… still uses a ridiculous amount of memory for the traffic I get (or got, before about 3am this morning). I have hosted dinky wordpress installs on Dreahost shared hosting for six years, and I have NEVER had problems until now. Something is screwy.

@DJ Allyn – Right on in so many ways. I once had a T1 go up and down daily for almost a month. At one point, it was down for a solid two weeks. Not only were my web services down, but the telephone and credit card services for my retail store. They were never clear as to why.
@Ulas C said @Ron “and Random Hypersexual are the only ones not complaining about this”.
I am not complaining and I am on DH VPS looking to upgrade to DH DS. Yes, DH clients are loosing money. Obviously, so is DH as down time causes clients to walk. The way I see it, the host and the client have a symbiotic relationship. So I am sure they are doing everything they can get things back to normal.
What I would recommend is a dedicated site / chat for Client Relations rather than Customer Support. Although it sounds a wee bit over the top in the feel good category, such a service would likely shift the weight from Technical Support so they could better spend their time addressing the issues at hand.
Maybe that way folk could feel more the comradely shared in a black out in the country side rather than the riots that take place in the city. Well, that and I could use a job during events like this cause I sure can’t work on my sites / software.

Website Hacked http://www.thickdame.com/

IP: 173.236.172.86
Decimal: 2917968982
Hostname: apache2-cabo.traceelliot.dreamhost.com
ISP: New Dream Network, LLC
Organization: New Dream Network, LLC

ps14049 still down.
Had a glimpse of the site about a half hour ago for about 10 seconds. Nothing since.

@pjk “This was caused by several software packages that were botched and/or removed by our packaging system.” is the answer to your “What was the issue?” question. Each VPS must be fixed manually. And there must be more than a thousand VPS’s.

@pjk Exactly the same here … moved to VPS ‘cos the shared server couldn’t handle 1000 pages/day, VPS crashes once a week as memory usage spikes over 300Mb for unknown reason (not traffic related).

Am uncomfortable with the fact they can’t restore an old server config for everyone – e.g. the last known to be working one! Someone at DH screwed up royally with an update methinks.

@Tuk

I’ve been a DreamHost customer since 2003. I have also had websites at roughly sixty other webhosting companies since 1996. I currently have websites at twelve other web hosting companies. So I kinda know the landscape here. DreamHost has its faults and limitations, but for the most part, they have less than any of the others in their class.

Five hundred plus comments here are NOT going to make any difference in fixing the issue they are having. A “small failure” can create a cascade of issues across a network, and they will have to deal with them exactly the way they are doing, whether there is one comment here or a thousand.

Some are asking why the shared servers got the attention first — it is because they all have the same exact configuration and can be given the same script to start up. The VPS and Dedicated servers are different. They each have their own configs and require more attention and time. DreamHost will try to get as many people up and running as fast as possible, so they went with the easy ones first. That would be the shared servers.

@Ulas C

I, too have seen much worse than this. It isn’t a case of “rationalizing”, it is just stating a fact. All of your whining and bellyaching about it isn’t going to make the process go any faster.

@Darren

YOU may not have experienced such a long interruption of service at DH, but others have, and I think that is kind of the point. MOST of the time you might have experienced a ripple or two in the service, but overall things have run fine for you. Same with everyone else here. MOST of the time your servers are up and running just fine. Sometimes they skip a beat and they go down for an hour or two. But that doesn’t happen very often, does it? (it might seem so to some, but I think if you were to chart every time you go down, you will find that it isn’t very often)

If you just glanced at this status site, you would think that DreamHost has a crapload of issues. They do, but they don’t effect everyone, all of the time. They operate a couple of datacenters full of servers and other gear, so you would expect there to be a constant number of machines that go down. It is why they hire datacenter technicians.

When it happens to someone else, we don’t really notice or even care. OUR machines are up and running, and that is what is important to US. If my machine went down and yours was still up and running, would you really care? Probably not, because you aren’t impacted by it.

But when our machine(s) are down, then the world is in the balance, and all hell is going to break loose if they aren’t fixed in the next three minutes (and yes, that is my foot tapping loudly while I wait…)

Some of us will even threaten to take our sites somewhere else. We’ll assume that the technicians in the datacenters are sitting around eating pizzas while your virtual Rome is burning. We will complain about inferior equipment and unskilled or inept technicians while we impatiently wait for them to restore our service.

In other words, we VENT, and LOUDLY.

But I still come back to the cases where people rely on Dreamhost or any of the other web hosting services in their class as a critical backbone to their business where, if the service went down, they are likely to lose THOUSANDS of dollars and might even put them out of business. If that were the case, then why aren’t they spending the extra money for a service that will NOT go down? Why are they willing to gamble their “thousands of dollars of revenue” on a service that only costs them around $50 – $100 a month depending on their usage? It doesn’t make sense.

@Ulas C (again)

“Interestingly enough, none of the “volunteer defenders of DH” so far revealed their “down” web sites, and none of their sites has gotten up that we know of. All of them are here and insisting on cover-up operation.”

My site (djallyn.org) itself DIDN’T “go down” because I am using Nginx instead of Apache. (you might want to consider that, since it uses less resources than Apache) But I still cannot access my site through FTP or SSH. So yes, I too am having problems at the moment.

I also manage a friend’s two domains on a shared server in which one domain is up and running and for some reason the other domain is “unreachable”.

I could whine, bitch and moan about how “bad” Dreamhost is along with everyone else here, but why? What will it accomplish? Will it make the Dreamhost people move any faster? Will it prevent another failure?

Here is a clue for you: for the most part, they don’t even read these comments. They are too busy trying to fix the problem.

ps37034 still down. Those who don’t understand the complaining don’t have customers/clients to worry about, and if you do, you won’t have them for long. We’ve been with DH since 2007, and now it appears it’s time to move. I’ve heard good things about RS and I’m currently exploring their features and plans. It’s too bad because I actually like DH, when it’s working. Moving is going to be a HUGE pain, but DH isn’t giving me a choice.

12 hours 1 min ago: Outage verified: We are actively looking into resolving it.

12 hours 29 mins ago: Outage first reported.

@ Andrews –Thank you for the comment. If you are paying, not getting what you paid for and still considering paying more money to them, that is your judgment. Everyone has a right to make irrational judgments. You can for sure use that right of yours. But why would you share this if that is not to find excuses for bad service, bad support, bad security (apparently hacked), bad of little bit of everything. This is business and should be subject to rational causation and reasoning, DH is not a sports team we are supposed to support and find excuses for them. It looks like you have not rational reasons but emotional reasons to back up them which is fine when you say it clearly. But what is going on here is beyond any reasonable limits to be tolerated within rational thinking in the setting of a company-customer relationship. It is a total and full failiure for a company and they will pay for this, just as we have been paid all day today.

DJ Allyn is one of them, obviously…Guest what ! His site is working, just as Random Homosexual’s porn site…

DJ Allyn said it all better than I can. People have no concept of how much of a mess IT can be.

I didn’t realize DH’s password woes made it into the news.

http://news.cnet.com/8301-1009_3-57363353-83/web-hosting-service-dreamhost-warns-users-of-password-hack/

This stuff just happens. Modern OSes are complex enough, and when you add the overhead for managed hosting of any kind, you’ve got a lot of complexity going on. DH has made some sensible decisions about sharing resources also but this means failures can hit more than individual machines. Nothing’s perfect. I’m logging off to do something constructive. I originally came here just to post obscene emosexual fan fiction.

Well done Dreamhost!

You’ve reached 11 hours of down-time on my websites (VPS). Just so you know, I’ve already set up all my domains on another host and brought my sites back online way before you could. I will be cancelling my account in a few days just so I can get all data off.

Thanks – member for 5 years.

OK so I’ve been down the entire day on ps17367. My index page worked for a while, now that’s gone as well. Is Dreamhost going to give us credits?

Random Emosexual Says:
DJ Allyn said it all better than I can. People have no concept of how much of a mess IT can be.

@Condom Homosexual, :) Of course he did, man, he is sitting next you, doesn’t he?
Well done you guys, you are doing better job here then your tech team.

@Ulas Ah I see. And yea, I’ll admit I am a bit upset. I moved my more “important” sites to VPS to prevent things like this from happening. Yet my fun/hobby sites on shared servers are fine and dandy (maybe I shouldn’t complain about that, I’m glad those have been up lol).

I guess it’s taking a while because they have to do everything manually for VPS.. yikes.

Both of my sites are still down, ps52131.

Come on. 12 hours down on vps!

Why fix shared first??????

13 hours 30 minutes, down…

January 29th, 2012 at 3:24 pmUnhappy Dreamhosters Says:

ps47664 is still down..!!!!

@Android Phone

“… moved to VPS ‘cos the shared server couldn’t handle 1000 pages/day, VPS crashes once a week as memory usage spikes over 300Mb for unknown reason (not traffic related).”

300mb is the minimum for your server. It usually requires a minimum of 150mb just to operate without ANY traffic. That would only leave you 150mb left over to serve pages. It is exactly the reason a shared server could not handle your traffic.

As I’ve pointed out above, if you are running Apache, you are probably using some resources that you don’t see in that nifty little graph provided. That graph might indicate you are using a certain level, but it isn’t taking into consideration your overhead. Those spikes you are talking about might not even show up in that graph.

I have a suggestion you (and anyone else here) might want to check out. Google JJ’s Memory Manager. It was written by one of the guys at Dreamhost on his personal site, Gimmesoda.com. You can download the program and install it on your server. It’s purpose will to adjust your resources as needed.

I have been running this on my site now for about two months and it has been working flawlessly. I used to have the problems you are having now, and now I don’t.

I’m glad to seee that I’m not the only one with VPS memory problems on my WordPress.

I don’t even have a very high traffic site, yet i”m constanlty crashing or running out of memory.

Does anyone have recommendations for where I should switch away to?

ps14342 server is still down? Can you guys tell me how long it will take to be back?

What’s going on!!! Did dreamhost moved to Fukushima??? Where are the happy happy admins to solve our unhappy servers? This datacenter is not trustable any more.. :(

12 hours down, still no access to my site

55174 is still down

DJ Allyn Says:
“I’ve been a DreamHost customer since 2003. I have also had websites at roughly sixty other webhosting companies since 1996. I currently have websites at twelve other web hosting companies. So I kinda know the landscape here. DreamHost has its faults and limitations, but for the most part, they have less than any of the others in their class.”

Perfect example of some of the tactics that these secret moderators and tension deflaters use:

1.Claiming some backround on the issue – “I know this stuff !” narrative…

2.Trying to convince the frustrated public that “things could have been worse”. They do it by using some tech terms and telling some worse disaster stories

3.Following this, they conclude on a very basic idea : “Do not leave, this is okay and possible”.

We have seen some of these folks today among these 600 messages. Ron, Condom Homosexual were flushed out early. SpiderSilk made an attempt and now DJ Ally pitches in writing even longer pages to convince the public that “things are okay, do not worry, keep bending forward”

The reality is, today is a bad day for the company and they need help. I don’t care if they were hacked, robbed, or failed. I care about the service I get for the money I pay. This is simple.

So you suck, including your amateur public agents here. Even they could have done a better job, most of them turned out to be sheer idiots.

@pjk Thank GOD it’s not just me with this:

“Also, it seems like other people were having the RAM issues, right? My (WordPress) site’s traffic never went past 600 page views/day, and I had to move it to a VPS because it was getting its scripts killed on the shared server for exceeding memory allotment, and on the VPS it was spiking past 400MB in the middle of the night, with no traffic. I killed all plugins, optimized WordPress by enabling the cache and whatnot… still uses a ridiculous amount of memory for the traffic I get (or got, before about 3am this morning). I have hosted dinky wordpress installs on Dreahost shared hosting for six years, and I have NEVER had problems until now. Something is screwy.”

I was told to get a VPS for the same reason, and will show huge memory spikes inconsistent with usage while the red line (the top one on the graph) goes to zero. Which I’ve written them repeatedly to ask about (after upping my memory *again* and *again*….like they keep telling me to, even though I’ve got nothing but a mostly-dinky WP install that isn’t getting anywhere near enough traffic to require 700MB. Nobody will tell me what the spikes are from (since it ain’t traffic), and only ONE person admitted it could be their server.

I’ve been with DH since 1996. Almost 17 years. It’s only in the past year or so that it’s become nearly untenable, and today’s outage may just be the last straw. I was already looking at a few other hosting options due to the memory runaround, but this kind of crazy outage that I’m paying way more for than I should be? I think this is the limit for me.

still down

@UC

Yes, my PERSONAL site is up and running — kinda. You can see it. But I cannot ACCESS it via FTP or Shell. I also have other sites on that server who belong to “clients” who also cannot access their sites, but can see them. So I am NOT really up and running, for all intents and purposes.

And yes, people DO have the right to be irrational. All I am trying to do is put it all in perspective, and making the observation that people who use web hosting companies like Dreamhost et al for their enterprise level requirements are taking a serious chance. Go to Rackspace or some other enterprise-level web host if your requirements are so critical. It would be like Amazon.com using DreamHost for it’s operations.

I don’t like the fact that I haven’t been able to get to my sites on DH all day either. I do have client’s with sites on servers at DH also, and have gotten a few emails about it myself. I have answered them pretty much the same way I am explaining things here — Downtime happens and they are working on the problem. They will get things up and running as fast as they can, and not a minute sooner.

It seems like a lot of people jump to the conclusion that DreamHost doesn’t care that their service is down, or that they are running a shoddy service.

mg Says:
I’ve been with DH since 1996. Almost 17 years. It’s only in the past year or so that it’s become nearly untenable, and today’s outage may just be the last straw. I was already looking at a few other hosting options due to the memory runaround, but this kind of crazy outage that I’m paying way more for than I should be? I think this is the limit for me”

Clear enough. I have been with DH for 3 years now and I am done.

ps14049 still down.

I’ve been monitoring this like crazy all day. VPS is still down, but I have access to FTP and HTML is rendering, so I created temporary index.html files and folders for all key links I have out there. Rather than ranting, let’s collectively share our solutions so as to solve the problem rather than just wait idly. These things happen, my question is how do you protect against a long outage such as this without breaking the bank? Does anyone have a good link to instructions for setting up a backup site on a separate server?

ps38523 still down, incredible.

My dedicated server is still down. Still getting the 503 errors and have lost about $400 in business so far today. How far out are we from a fix?

@DJ Allyn –Simple question, you are a customer, right? If that is the case, you are a losing along with all of us now. What is your motive of writing 700 word personal statements in order to find excuses for this disgusted service? Why the F do you care?

And, who said that they are not doing their best? And who cares if they are or thet aren’t. Bottom line , my all sites have been down for 14 hours now almost. Why would I, or would you, care how much they try? Are you a fan of DH Sports Team? Or a DH customer?

Thanks Jeff; I too have redirected my own traffic, recreated backup sub’s and 2nd server hosted. Fast and Easy, instead of just crying about it.

ps17440 is still down

@ Jon Gordon

can you explain how I can do it?

ps48770 down , approaching 14 hours.

DO we need to complain personally to DH for a credit or will they just issue one to us? I think we all deserve a comp at this point.

12 hours. All my sites down. Finally back up. I really don’t mean to pile on. I love Dreamhost and I know you guys are doing your best.

BUT: I’ve been recommending DH to clients as the go-to hosting. I can’t do that anymore. Even if they have to pay more, I’ll tell that for maximum uptime, they need to go to . . . Well, fill in the blank.

Uptime is crucial. You know that.

I’m sad and disappointed.

12 hours 38 mins ago: Outage verified: We are actively looking into resolving it.

13 hours 5 mins ago: Outage first reported.

Okay, one of my FTP accounts is working.

@UC

You should already have a local backup first of all, if not, see if you can backup all your remote sites locally (That is if you have FTP access; I know mine was down for awhile). Once done, you can either make backup sub-domains on DH or use a 2nd hosting server somewhere else. Redirect you current DH domains to either the new backup sub-domains you created or to the new hosting area. Done.

My high end client is fit to be tied. He wants to move back to Godaddy. Godaddy sucks, but you guys have had soooo many issues along with the recent security breach that he would rather go back to a slow over sold hosting company! He said he’s losing faith in my judgement on hosting companies. F**K!

Update Jan 29th, 3:45pm PST: Not very much to update from the last post. There’s still a handful of dedicated servers that we’re still fixing MySQl services on. There’s only a few shared hosting servers left that need to have FTP fixed on them. VPS servers continue to receive the most of our admin teams attention and we’re getting those configured and restarted as quickly as we can. We’ll update this post again shortly with more info. -dan w

@Jon Gordon – Thx, what is your new choice of hosting company?

@UC

You give me WAAAY too much credit.

I am sitting here bored — probably like you, waiting for DreamHost to get things back up and running. If MY stuff was up and running, I wouldn’t even be here right now. I only show up here when MY stuff isn’t working. I am kinda selfish that way.

I mean, I could use my Comcast voice here, but what would that accomplish? (anyone who has ever had to deal with Comcrap on the phone probably knows what I am talking about) It isn’t going to get my stuff up and running any sooner. I could threaten to leave, except that I have almost a TB of music on my personal site alone. I could badmouth DreamHost for being inept, but are they really? Or is this just a case where sh*t happens and they are working on it.

VPS MEMORY ISSUES

Seriously, check into JJ’s Memory Manager. Consider switching from Apache to Nginx. I solved all of my memory issues doing that, I went from using 1850mb down to 500mb — and I am serving a LOT of music and video files.

I used to be confused by the stupid graph on the Manage Resources page for my VPS. Right now it shows that I am using about 200mb, but that isn’t accurate. JJ’s Memory Manager is showing that I am actually using about 480mb with 20mb left available. If I get a traffic spike that requires more memory, the manager will increase it automatically, and drop it back down when it isn’t needed. It is pretty slick.

DJ Allyn: My memory issue is on the MySQL VPS.. not my web VPS.

I can’t switch anything, that is all managed by DreamHost.

Yet for some reason, my MySQL VPS, is using 400+ MB of ram, while only using .1% of it’s CPU.. and is pretty much only running 1 small VBulletin forum, that is closed to the public.

The odd thing is that their rollout tests on their rollout test servers don’t test for something as basic as a simple test.php file with no mysql connectivity or an ftp connection. We are down to one site on dreamhost so aren’t worried about it, but seems odd. If they were better communicators, I would ask about their test build process and QA.

It would probably be much cheaper for dreamhost to improve their testing infrastructure than have a bunch of sys admins running around on a sunday.

For reference: My Web VPS RARELY goes over 100mb usage…

But somehow.. my MySQL VPS is over 400mb constantly..

Something is wrong, and it isn’t my fault.

@DJ Allyn– Okay, what can I say. S..T happens then. FML. Seriously, despite all my screaming all day, I have loved DH so far but can’t tolerate this one. Need some advice on moving. I have never done that. Understand that DH is better priced and some of the things happening here have been attributed to their “good pricing”. F that. I am willing to pay more not to go thru this one more time.

“I used to be confused by the stupid graph on the Manage Resources page for my VPS.

Same for me. I have never had a good understanding of the graph, all I know is DH has kept bugging me to pay more.

After 14 hours of patience, I cannot wait any more before I ask, what is going on at DH? My site has been penalized by Google because of too many downtimes. That is what my Google support ticket said.

14 hours!

13 hours and counting – still no change VPS now pingable but no http / web services running

HTTP of ps54151 and ps54152 still down since the beginning. However I can now use FTP. I’m downloading all my sites right now and as soon as the phpmyadmin work I will download the databases and move as fast as I can. I have never seen anything like this. I’ve been working with hosting companies such as MediaTemple, Easyspace, 1and1, HostingZoom and many others for years and moved away from them for much less downtime.

I’ve been with DH since 2007 and always been impressed by their customer’s support. However the current downtime is getting ridiculous. They should have a plan B setup for such a disaster scenario… Objectively I am very disappointed and I’m not whining. Just realising that DH has completely and utterly failed to deliver a good service.

@julianna oh wow.. seriously? Google is going to ding us because DH can’t get their crap together. FML

Signing up at hostgator right now… Shame on you Dreamhost. Shame on you. That’ll teach me to signup for a dedicated server at the beginning of a new year.

@UC

I like http://mediatemple.net/ as an alternative. It isn’t cheap, but it’s still online, if you get my drift.

A 14-hour outage is bad. No ETR after 14 hours is inexcusable.

Same here Julianna…so many of my “broken links” are from DH downtime.

DH, dan w – what exactly happened here?

Saying it was a config problem isn’t saying much.

Will your VPS customers be getting refunds of any kind. Losing hundreds, possibly thousands of dollars, from today’s full day of downtime.

Better host suggestions, anyone? Need to start looking for better hosting.

Everything is still down for me. This is horrible. I can’t begin to imagine the overtime hours all of the techs are going to put in on this.

New SLA on DreamHost.. 99.9% Down..

OK…i have ftp access but 3 of my sites are are still down. going on 13 hours. whats the best way to save everything so my transition to a new host will be seamless. thanks in advance for any responses.

13 hours 6 mins ago: Outage verified: We are actively looking into resolving it.

13 hours 34 mins ago: Outage first reported.

Could some one confirm if one of their VPS is back up? Or could Dreamhost just say all our VPS are down.

@anon

That sounds like you have an issue with one or all of your databases. Have you tried to optimize or repair them? Are you running some kind of strange characterset?

Why aren’t you able to set the resources on your SQL VPS?

For reference: My Web VPS RARELY goes over 100mb usage…

But somehow.. my MySQL VPS is over 400mb constantly..

Something is wrong, and it isn’t my fault.

First of all, as I said before, relying on the graphs on the “Manage Resources” page is useless. It is a HUGE complaint I’ve had with DH for a while now. That was how I found out about JJ’s Memory Manager — one of the guys there turned me on to it after an especially long rant I had in a support ticket. He also told me that DH is working on a different interface that will more accurately show your actual resources.

Secondly, just running without any kind of load, your server is going to require 150mb just to run. That is why the graph they show isn’t going to be accurate at all.

Yes, it does sound like something is wrong. Whether it is anyone’s “fault” or not, that isn’t the problem. What you are looking for is a solution. I don’t know what your database is running, but it does sound like it is being seriously taxed for some reason. Especially since you indicate your web server doesn’t seem to be operating at all. It sounds like you signed up for the VPS and forgot to transfer your domain over to it. I did that once — I went for three weeks wondering why I wasn’t registering anything on the resources graph. duh…

Try to repair and optimize your database by going to phpMyAdmin.

yesss…13 hours 30 minutes, down… it’s 2012. this is unacceptable.

my VPS is still down :(

this reaaaalllllyyyyy sucks.

14 hours down, a big big big #FAIL. Thank you Dreamhost.

DreamHost sucks. End of story.

Still down. No answer. Nothing.

I would tell everyone to run. 15 hours down and still no info.

Don’t host with them. They will kill your business.

Gotta love the ‘I’m outa here!’ comments. You can all jump ship and let the those that stay have snappier VMs. If you haven’t experienced an infrastructure meltdown, it’s probably because you don’t manage any significant infrastructure.

Still Down. This so F/ed UP.

All my regular sites are back up, but the site on VPS, the one I pay extra for and want to be extra-reliable, is not. I’ve been with DH for years, and this last year has been unusually flaky. I hate to jump ship, but I’m starting to wonder.

It would be nice if DH could forfeit details about what caused this massive outage.

What the hell is going on? We’ve lost an entire weekend day, and our business is built upon weekend traffic. We heard nothing in advance (to my knowledge), and we’ve had customers asking us all day what the hell is going on… “We’re working on it…” sounds pretty lame, but unfortunately, that’s all we can tell them, because it’s all you can tell us.

What could have happened? Their entire hosting must have been defaced, what could explain such a disaster…They do not even tell the truth.

“This was caused by several software packages that were botched and/or removed by our packaging system. This was a new, unexpected bug that is unrelated to any of the outages earlier in the week”

This is a lie. We are not stupid. Nothing like this could have caused the longest downtime in the history of the company. Tell us the truth !

We demand the truth.

Letting you Dreamhost guys know that short of receiving a free pony, I’ll be moving my website elsewhere.

@UC

Understand that DH is better priced and some of the things happening here have been attributed to their “good pricing”. F that. I am willing to pay more not to go thru this one more time.

Okay. I can give you some insight there. BlueHost is good, but I would go with their “Pro” package at $19.95 a month. It will give you the equivalent of a VPS running at around 450mb. I wouldn’t try running too many sites on one account though. Also, the pricing is a minumum of 12 months up front.

Media Temple is pretty good. I have had some problems in the past with their “grids”.

IXWebhosting is another one that is pretty good, but they do have their moments….

If you want a SOLID enterprise level webhost for a reasonable amount of money, I would personally go with SPRY.COM. I have over thirty sites there and have had very little problems that can’t be fixed with a telephone call.

I want to hear from the Dreamhost CEO now.

So you guy’s don’t have backups? Didn’t prepare for this? That’s your own fault.

DJ A: I a can assure you my web site is running on the web server, using less than 100mb. It has been unchanged for about 6 years… but for some odd reason,after years of the DB server only using minimal memory, the MySQL suddenly starting using 400mb+..

I can’t install a memory manager on the MySQL VPS.. I can only use DH’s memory graphs. You have absolutely no access to your MySQL VPS other than through database tools.

I’ve run all the DB fix/repair scripts I possible can.. but this is a database with maybe 100,000 total records across all tables… so again..

My website uses 100mb… the MySQL database for this site.. 400mb…

Something is wrong, and DH needs to fix it. I’m not paying $25.00 a month just for a MySQL database for a web site with no traffic.

Are you guys kidding me…….. still down? It’s been 15 hours. Hope you won’t be missing my business when you bring your servers back online because I’ll be switching hosts immediately. Absolutely horrible service, and horrible hosting.

And I don’t want to hear another person tell me not to use the fucking graphs provided by DH.

They either need to fix their graphs, or give away all the free memory in the world, because it’s patently ridiculous to tell your customers to ignore the only tool you provide them for memory usage stats, right on the page where you SET YOUR RESOURCES.

And again.. I have no other way of monitoring my MySQL servers memory usage.. I can’t SSH into it, I can’t do anything with it but trust DH’s graph.

Where did I say I couldn’t change the resources? My point is I’m done PAYING for these extra resources I clearly don’t need.

January 29th, 2012 at 4:36 pmSome data removed Says:

Very bad news, it seems that the data of sql tables of major part of vps servers have been lost!!! However DH will restore the data from a stored backups of the last month…

DJ Allyn: Which VPS package from spry.com that is handling over 30 sites?

This is getting beyond ridiculous for downtime.. people pay for VPS to avoid this stuff.

An hour without an update. WTF

@DJ allyn — Thanks !

questions :
1Just looking mediatemple, liked it, appears okay, what are grids?
2SPRY.com – is it the best among you mentioned?
3.Will be running multiple busy forum sites with 2000 active members, 30-40 people online at any given time. Have been very confused with DH graphs, could not understand it fully, so I do not know what I need really. Do I need a VPS really? If yes, what is an “above average” service price for this kind of a project?

Still down, still down…

OpenStack Crowbar infrastructure-as-a-service cloud solution / automated hosting solution issue?

I think the transition isn’t going so well.

is not a subset, Every VPS under Dreamhost roof is down.

I have an old VPS account that is up.

I also have a separate new VPS that is down.

Sunday is my busiest traffic day, and after 8 months, I am finally seeing an uptick in my traffic. This is a real setback.

What’s really cool is that my cheap shared hosting was working fine all day and I decided to “upgrade” to VPS. Seems I upgraded to down service. Yay!

Our business has lost an incredible amount of potential leads today from this. I have never in my 15 year span of developing websites seen a host go down for an entire day. Updates or not. An entire day down.

It is already to late I am now forced to move to a new host.

Anyone hear anything about Amazon web hosting?

ya what is the truth here. what is the real deal? they going under?

Mediatemple vs Spry

Any comments? I am about to choose one right now.

13 hours 43 mins ago: Outage verified: We are actively looking into resolving it.

14 hours 11 mins ago: Outage first reported.

http://www.kimsufi.it
From OVH France in Italy… 14.99 for a CELERON 2GB ram 1TB disk DEDICATED! Better than vps

I’m moving to it

Well, thanks for motivating me to finally move all my WP sites to AWS. After the recent hacking issues, and this sh*t-show, it’s definitely time to move.

Gee, right now I had to cancel multiple meetings with clients, since every commercial site of mine has been down nearly 18hrs, and now that it’s Sunday evening I have no choice but to cancel my Monday appointments too. (After all, they’re not even *pretending* they’ll have sites back online by morning… and even if they did, how could I possibly believe it?)

Wow.

I continue to be amazed by the futt-buckery.

who’s gonna start “class-action-against-dreamhost” dot com?

23 updates over 14 hours of downtime – and still no real info about what happened.

Critical Announcement! Please Read!
We’re currently experiencing the following issues:
– webmail login issues: status post
– Web, SSH and FTP issues on some shared, VPS and dedicated servers: status post
– If you haven’t changed your FTP/SSH password since 1/21, please do so now, as your old one won’t work any more:status post
– If your email password keeps reverting to the old one, we apologize. This is being investigated.
We apologize for all inconvenience all this is causing. (posted 3 hours 51 mins ago)

This is ridiculous. It would be one thing if these types of outrages weren’t so common with DH. Some of us have livelihoods depending on our websites and a full day down is unacceptable. I lost far more money today than I’m currently paying you guys per month.

Will be switching hosts by the end of the week.

As an aside, I gotta say, it’s mighty kind of them to let us all share advice on which competitors we should switch to, on their own site…

As another aside, it’s interesting to note that whatever experimental Shiite they installed that skreued up all our systems, they somehow managed to maintain continuity on the server that we’re using for these comments they’re ignoring…

HEY, DREAMHOST!!! CAN I HAVE MY WEBSITES HOSTED ON THE SAME SYSTEM YOU USE TO TELL EVERYONE THEIR ACCOUNTS ARE F’ED UP???

It seems to be the only thing they can reliably keep online…

This is brutal.

@UC said: “DJ Allyn is one of them, obviously…”
Of @DJ Allyn, @ UC said: “Perfect example of some of the tactics that these secret moderators…”
@UC went on to list: Ron, Condom Homosexual, SpiderSilk, as members of the conspiracy. This brings the list of conspirators to five. I feel left out.
As one of other long term DH customers with broader than DH experience, I feel very left out that I have not been named in the conspiracy. Maybe its because there are two folk posting as Andrew. Will change how I sign my posts to Andrew John and maybe I will get a little more respect from @UC.
@Mg said of WordPress: “…still uses a ridiculous amount of memory for the traffic I get…”.
@MovingAway and others joined @Mg in commenting on WordPress installs using too much memory and resources.
Wordpress has grown into a resource hog for what it is usually used for. In the past, most shared hosts forbid WordPress MU installs (including DH) but welcomed WordPress. Now that the two branches have merged, it –seems- that even if you do not enable multi-site or multi-domain features; the thing is a hog if all you want it for is blogging and content management on a single site. I said it –seems- like a hog because it doesn’t have to be and you can change lighten the load by changing the config file.

WordPress DH Hint – Delete the config file created by the DH installer robot. Then run wordpress to create the new config file. I –think- the installer is turning on or failing to turn off some of the bells and whistles that wordpress does when it creates its own. You could also write your own from scratch, but that is a royal pain in the rear.

Brutal is right.

@UC

“This is a lie. We are not stupid. Nothing like this could have caused the longest downtime in the history of the company. Tell us the truth !”

Why would you consider them “lying”? You make an assumption that they did this to you on purpose or that they are somehow out to screw you. Don’t you think they would rather be doing something else with their time right now instead of trying to get everyone back up and running?

I just got my FTP/Shell back a few moments ago, but now I cannot reach phpMyAdmin. I get a 404 message. Should I wind up my Comcast voice yet? I mean, I can get quite screachy with it when I have to. You should also hear my “You People” speech.

Where did I say I couldn’t change the resources? My point is I’m done PAYING for these extra resources I clearly don’t need.

Then go somewhere else. Obviously, you have a problem within your database. It is YOUR data, not your web host’s. Try this: disconnect your current database and create a new Vbulletin database. See what your resources are doing then. If things are calmed down, then import the data over from the old database.

Where did I say I couldn’t change the resources? My point is I’m done PAYING for these extra resources I clearly don’t need.

Apparently you DO need those extra resources — THAT is the point. What you need to do is make it so that you DON’T need those resources. If that requires you to create a smaller database, then that is what you will need to do. 100,000 records is fairly large for a database, and the more you add, the more resources you will need to filter through them.

I’ve taken large WordPress sites and broken them up into years because their databases got so large that it slowed everything down. Maybe you should consider doing that with your forum. Break it up into smaller chunks. Six years worth of data in one spot might be a bit much.

It is just a suggestion.

@Adam Gerts

DJ Allyn: Which VPS package from spry.com that is handling over 30 sites?

I actually have four separate accounts at Spry. I have two of the 2048 accounts and one each of the 512 and 1024 accounts. I would say that the 1024 account is probably plenty for someone who is running an ecommerce site with a LOT of traffic at one time.

@UC

1Just looking mediatemple, liked it, appears okay, what are grids?

It is similar to “cloud computing” but not quite. It is basically redundant hosting, but if a grid goes down, it may make your site dead to a particular geographical area. There are some issues with different grids not updating as they should, so I’ve experienced different pages being served until they all catch up.

I would go with Spry. You have FAR more control, you can even give your clients their own cpanel if you want.

still down
now 15 hours.
Thank God, FTP’s are working, downloading my last backups…

No update for almost 2 hours now.

Shame on you DH, shame on you. Close this business and go home…

And now, who has seen worse than this now? Come forward…

DJ Allyn: “100,000 records is fairly large for a database”

This is just laughable…

Brutal indeed.

@DJ Allyn,
“Why would you consider them “lying”? You make an assumption that they did this to you on purpose or that they are somehow out to screw you. Don’t you think they would rather be doing something else with their time right now instead of trying to get everyone back up and running?”

Because this disaster need more explanation then this. My all sites were hacked 2 weeks ago. And now we have evidence that (see above messages) some of their sites were defaced. This might explain why they shut down all the system and put a “maintanence screen” while Apache is running. They would not want to say that they are hacked. What kind of a sotware update take the whole day to install? Something is very unusual here, and it is so obvious.

which one is more upscale and reliable company?

Mediatemple or Spry

I will go with your comments here.

hey… spread the word via twitter, #baildreamhost :)

Wonder what DH CEO @DreamHostSimon is doing right now??

DJ Allyn: “100,000 records is fairly large for a database”

This is just laughable…

Hey, don’t shoot the messenger. You are trying to squeeze an elephant into a shoebox and not liking the results. The fact is, you are using more resources than you are willing to pay for, and that is understandable.

I gave you a suggestion that would help you achieve that goal. Break up your database into smaller chunks — by the year — and work with that. That way you can use less resources by serving up smaller amounts of data.

Just saw the comment, sorry.

I am going with Spray then..

BTW
still down, everything…

hachiman is down!

BTW, whoever said something about “you’re trying to squeeze an elephant into a shoebox”… I gotta say, the dozen-or-so sites I’ve got hosted up here are all quite small. And the level of VPS service I’m paying for is about 4x the peak usage I *ever* see.

And, yes, it’s perfectly reasonable to have a database with 10^9 records, let alone 10^6… the problem is a host that can’t figure out how to hire sysadmins who can wipe their own butts… or how to at least hire an honest customer relations dept. Not some guy with 100k db records.

MY web server ps54241 & mysql server psmysql54279 STILL DOWN
Thank god it looks like emails were not effected as I left them on the shared hosting.
God I sell to businesses, I had a period of a week where I was not well.. then I get well enough to continue and Bang Christmas hits and all business owners vanish for 3 weeks.. then I spend a week in hospital.. then I get well enough to continue and Bang Australia Day public Holiday and then My VPS goes down all day and still not resolved.. What is next a plague of locusts. I think the world is out to get me.. Bad timing DH… just bad bad timing…
NO FTP – NO MYSQL – NO WEBSITES all VPS SERVICE IS 100% DOWN FOR ME..
Imagine if this continues throughout tomorrow… OMG that would just be final nail in the coffin.

January 29th, 2012 at 5:19 pmUnhappy Dreamhosters Says:

My VPS is still down and the billing still running. I need a compliment for this outage :(

The Maya calendar was tight.

aCompliment for outage ?…..ha ha… okies.. “thats a very nice outage you have there”

The Maya calendar was right!

NO FTP – NO MYSQL – NO WEBSITES all VPS SERVICE IS 100% DOWN FOR ME!

So, clients of DH, once again, this is why you back your clients data up every day and have a backup host. Nevermind the the downtime, just be prepaired for things like this.

@UC

Go with Spry. Start with the 1024 plan and you can change it later as you see fit.

As far as the rest? It wasn’t wide spread, it didn’t effect their entire network. And it WAS basically an Apache issue, because it didn’t take my personal site down. (Nginx was what saved me)

Sites get hacked all of the time, regardless of where you are hosted. Defacement is a common problem everywhere, and has nothing to do with a particular host. The fact that DH detected a problem a week ago and went through the forced password change only goes to show that they immediately took action as soon as their software detected a POSSIBLE alarm.

Nothing is safe. Like a lock is designed to keep honest people honest, there are no SURE THINGS when it comes to password security. For every security protocol, there is a security failure waiting in the wings. Look at the Anonymous group of hackers. Given time and inclination, anyone can breach anything that has a dynamic password. If you don’t change your online banking password on a regular basis you are asking for problems there also.

I am sure that DH is doing everything it can to prevent breeches. But even the most vigilant measures can be thwarted from time-to-time.

I hope you have a good experience at Spry. I think you will like it.

VPS ps24490 websites are still down, but my FTP is back and I’m backing up my stuff. Please please get it up and running. I hope you can understand the extend of your outage!

@Joe – I saw your explanation above, and it made sense. To switch to the backup server, do you just create new DNS A-Records? Do you register domains at Dreamhost?

14 hours 20 mins ago: Outage verified: We are actively looking into resolving it.

14 hours 47 mins ago: Outage first reported.

No VPS.. no http://FTP..

500 Sorry, no server available to handle request on apache2-twang.trafficante.dreamhost.com

DJ Allyn: Except what you are saying is patently ridiculous. The database is TINY..

I have another VPS, with several million records in it’s main database.. uses less resources.

You are trying way too hard to sound smart. 100,000 record database needing to be broken into pieces? LOL

Wow i’m glad that it happen before the lauch of my project. Lots of money and time invested in that projet and now I see from the many userin this page that DH isn’t a reliable solution since it happen a lot. It’s clear now that i’ll move to so a more serious business.
I’ll make good use of my google+ and facebook to prevent other people from doing the same error.

@Aaron Sylvan

Then leave already. Go find that magical web host with the unicorn that farts multi-colored sparkles.

It is funny how some people can never take responsibility for their own stuff. If you run wordpress sites with thirty plugins, hundreds of pictures, and sites with unruly scripts, what do you expect? If you have a large database and are trying to get by with a small amount of resources, then again, what the hell do you expect?

INCREASE YOUR RESOURCES! Or just cut back on your bloat. Go through your code. Take out some of those plugins. Break up your database.

Or MOVE. Find a host that you think you will be comfortable with.

Is InMotion any good? I’m seriously considering switching, but so many of the hosting review sites look like shill “fake review” sites.

DreamHost taking customer service advice from the airlines…most valuable customers get worst service….fixing the VPS last?!?!?! Doesn’t it make sense A. That the VPS customers rely on the service more than the shared, and B. Are more likely to be internet influencers???

anybody uses knownhost ?
I have. Very fast cuctomer service, respond time under 2 hours , most of the time in 30 minutes, friendly , will install anything you want, unlike DH..
1200MB RAM for $45
960MB RAM only $35
Been with them 4 months and never down.

January 29th, 2012 at 5:44 pmbuythiscomputer Says:

***************************************************************************************************
***************************************************************************************************

WE, VPS CUSTOMERS, PAY EXTRA, SO WE WANT THAT DREAMHOST REPAIR OUR SERVERS FASTER AND REPLY TO OUR SUPPORT TICKET FASTER ALSO.

You treat me like shit i pay you like shit !

***************************************************************************************************
***************************************************************************************************

At last FTP up and running. I need SQL

800 comments and counting. I am leaving dreamhost. I have been here since 2006.

One VPS up.

The other still down.

Just FYI, done arguing with DJLame.

For users with unacessible http://FTP.. Try to use WINSCP over SSH on shared service.. worked for me.. the VPS still down..

My site just got back up recently. I suggest checking ‘top’ from shell command to see whether apache is running, and if not, go to the panel, switch to nginx as http server, and then switch back to apache.

WINSCP loggin in on VPS!!

DJ Alyn

Are you saying that this FUBAR is happening all the time when you are saying “sites get defaced all the time”.

I have been here all day, it has been 15 hours all my sites down. I have noticed at leas one person here continuously defending DH disaster. Now it happens to be the DJ Alyn. Please stop this. Stop this. This is unacceptable, universally UNACCEPTABLE. THERE IS NO HOSTING SERVICE IN THE WORLD WHICH GOES DOWN FOR ALMOST 24 HOURS AND STILL FIND AN EXPLANATIN FOR IT. NO EXCUSES. NO EXCUSES. THEY SHOULD HAVE HAD A BACK UP PLAN. THIS IS BEYOND ALL REASONABLE, PLAUSIBLE LIMITS. THIS IS IS F/ED UP BEYOND ALL RECOGNITION. THIS IS FUBAR. YOU ARE ANNOYING, YOU ARE ANNOYING PEOPLE, RON, CONDOM HOMOSEXUAL, DJ ALYN, AND OBVIOUSLY YOU HAVE SOME KIND OF PARTNERSHIP WITH DH TEAM.

DH ! INSTEAD OF COMING UP WITH AN HONEST EXPLANATION AND CUSTOMER SERVICE YOU ARE SENDING THESE IDIOTS TO CALM US, SHAME ON YOU DH, SHAME ON YOU !

14 hours 41 mins ago: Outage verified: We are actively looking into resolving it.

15 hours 9 mins ago: Outage first reported.

KNOWNHOST VPS:
1200MB RAM for $45/mo ($39/mo every mont with 15% coupon)
960MB RAM only $35/mo ( $29/mo every month with 15% coupon)
1472MB RAM $65/mo
1600MB RAM $85/mo
Been with them 4 months and never down.
Cpanel is $5/mo extra but with 15% discount coupn its practically free.

Interesting. DreamHostStatus.com is hosted with http://www.Linode.com.
You can check this by pinging DreamHostStatus. Checking them out now.

SFTP working to..

Have been waiting since 10.30am this morning its now 2am the next morning and guess what still no blog ….

@Gerg — they moved status over to another host entirely after some disaster a year or so ago that took down everything for a few hours. If I remember correctly, one of the main trunk cable thingies was all screwed up, and nobody could get to the panel or the status, which was then hosted on DH’s own servers. This one was created as a separate backup in case of DH-wide emergencies.

I could have quite a bit of that wrong, though. Server admin stuff makes my eyeballs bleed.

My sites must have come back on while I wrote that last post.

OMG, now FTP is down again. I hate you guys. Just trying to get my stuff, even that is not possible. What kind of a loser hosting company puts itself in such a pathetic situation !

This worked for me!! Thanks @acnetj

“acnetj Says:
My site just got back up recently. I suggest checking ‘top’ from shell command to see whether apache is running, and if not, go to the panel, switch to nginx as http server, and then switch back to apache.”

All my sites are up.

Is anyone else here having luck getting their sites back online?

@UC

I’m thinking the ALL CAPS are really effective, as are the pejorative comments about “loser hosting companies”. You’re really making some headway. Keep it up.

LOL it looks like @Ulas C’s site is still down, i guess when you are complainer who throws out their ps# you get pushed to the back of the line

My 50+/- sites are all back up. Have faith – it only took a little over 16 hours. Ugh.

:)) Looks like Ron is back, taking over from DJ Alyn, now with a bud Chris…Clear cut shift work here. Welcome Ron, how were things back there? Give us some valuable information…

Garbage in Garbage out, one idiot is on-call at all times

everything is still down for me…

All down for me also

huhu, my all website errors, they do not work. My Website is business, my customers are complaining.

Wishes the Dreamhost support team troubleshooting fastest.

I suggest checking ‘top’ from shell command to see whether apache is running, and if not, go to the panel, switch to nginx as http server, and then switch back to apache.”

did not work for me.. all still down

My ps had ssh for 20 minutes and apache is now back up and running. Looking forward to a full explanation with tech/software incompatibility references tomorrow before I decide on migrating.

All of my sites work, sucks to be you guys.

@UC
Err, no. I have many sites down, and am a DH paying VPS customer. I’m just sayin’, you’re gonna have a heart attack. It’s not worth it. Breath.

It says above

“Update Jan 29th, 1:40pm PST: The shared end of this outage should be resolved at this time. Both http and ftp services are up and running on all shared hosting servers”

My site is still down, im on a shared server

Im on sinistra

Ok. Another hour and fifteen mins without any info.

???????
Is ANYONE’S vps server up now that was down before.?????
??????

I just want to know if they are actually doing anything with vps server or if I should move my shit to a new host.

15 hours 23 mins ago: Outage verified: We are actively looking into resolving it.

15 hours 51 mins ago: Outage first reported.

SWITCHING TO OVH

How bout we al just shut up until they finish. 800+ comments wont get it done any faster. Geesh.

Just letting others know that my VPS did just come on-line. Took wayyyy too long but it does appear that they are getting them online.

still down, not good for business, every painful refresh shows a 503. Time to start stringing up some of these hackers or addressing their concerns, what ever the case I don’t like being collateral damage

Greg, what URL?

January 29th, 2012 at 6:41 pmCeeGee Says:
How bout we al just shut up until they finish. 800+ comments wont get it done any faster. Geesh.

——-
I just want some info. An eta. Some numbers like 400 of 2000 vps are fixed for something. Something to tell me thy are doing something. Right now for all I know it could be down for a week!!!

Starting tonight we are moving all of our hosting to the Amazon Cloud. This is unbelievable. 15 hours down so far. Looking forward to hosting with a more reliable host.

17 hours and all my database dependent pages are still down. At least my static pages work so I’m not losing 100% of my revenue for the day. This service has been worse and worse since I ‘upgraded’ to a VPS.

My site and Ftp are back up, but my site is terribly slow, similar to the past couple of days

15 hours 36 mins ago: Outage verified: We are actively looking into resolving it.

16 hours 4 mins ago: Outage first reported.

I’ve spent my day monitoring this and telling my people to hold on waiting the SCC on my VPS to come back online… 16 hours later I’m still telling the same thing… but I’m also switching to OVH: as soon as I can get my hands on my data I’m out of here :(

My VPS came up about 10 minutes ago.

@anon

Just FYI, done arguing with DJLame.

No wonder you are having problems. You don’t want to listen to anyone.

The thing about having a self-hosted website is that you have to have an inkling of what you are doing. DreamHost isn’t there to hold your hand and fix your individual problems. They are simply providing a server for you. What you do with your server is up to you to figure out. If you are running something that is bloated, then it is up to you to figure out where that is.

You have a problem. You just don’t like the solution — which is another problem. You have a database taking up too many resources for your minimum setting and you refuse to increase the resources. I gave you some suggestions to try and all you had were excuses. It couldn’t possibly be your data, it HAS to be the server.

It is really quite simple to test that theory out, genius. Order up another VPS SQL server and move your database to it. Then turn off the original one. If it was the server, you will have solved your own problem. If you are still having the problem, then guess what — it is your database. Duh.

Or you can continue to whine, bitch and moan.

@Gerg

Interesting. DreamHostStatus.com is hosted with http://www.Linode.com.
You can check this by pinging DreamHostStatus. Checking them out now.

There is a logical explanation for it. What if the entire Dreamhost network went down. How would they be able to tell you what was happening, unless they were hosting this on another network?

@UC

Would DH send someone like me to tell you to go f*ck yourself? Or tell you another place you can take your websites to? That would be a great customer service exercise, wouldn’t it?

Sh*t happens, and it is unfair when it happens to you. But tossing out blame with no substantiation is downright silly. DH had their hands full on this one, and all you could do is bitch about it and accuse them of covering up something with no facts, just your fertile imagination.

Your site(s) got hacked. It happens. Even the NSA servers get hacked, and I am pretty sure that they aren’t hosted on DreamHost. Change your passwords every now and then. Be proactive. Don’t assume that things are secure because NOTHING is secure enough. You have some responsibility in this also.

And if you don’t have a complete backup of your own stuff, then you really are asking for trouble. There is no excuse for you not to have a backup.

For all of you who are leaving:

Buh Bye Now. Sheese.

Would be helpful if we could see PROGRESS here – i.e. – We have X servers out of Y remaining to be fixed. We have to do X,Y, and Z to get things fixed – its estimated that X will take ‘this’ long, Y will take ‘this’ long, and Z will take ‘this’ long.

Not a lot of confidence being inspired here.

@ Jon – Acrius

Is Amazon Cloud a hosting service? Does that provide VPS and is it better than Spry or MediaTemple?

this is so ridiculous. i hope you are planning on refunding VPS users for the money they pay considering its been down for 16 + hours.

January 29th, 2012 at 7:04 pmSteve Olejarczyk Says:

VPS back briefly. Now http connection resetting.

@Kevin Davidson – Thank you for letting us know your VPS is back up and running. It gives me hope they are progressing through fixing the issue and they are getting them up although mine still isn’t.

DJ Allyn: I’m a professional software developer who is well trained in server administration as well. You are here trying to claim a 100,000 record database is large!

You are being a complete tool at this point, misrepresenting your own ability to try to speak to people’s technical problems.

You also keep ignoring that I have absolutely no control over the VPS hosting MySQL, only the MySQL db itself.

I just checked, and my ENTIRE DATABASE is less than 30mb.. the largest table is less than 15mb.

So now, explain to me why my VPS that was using < 100mb of memory, suddenly went up to 400mb usage last December out of nowhere, unless.. get this.. DreamHost is having some problem on the MySQL server causing this memory issue?

DJ Allyn: Also just ran a repair/optimize on all tables.. all were up to date.. nothing needed.. because I do that regularly.. and as I said.. this database, is BARELY USED. Maybe 40 new records a day.. max, written to it.

Dj Alyn – Behaviors have motives. If you are a customer who is trying to rationalize 24 hours of downtime by giving “worse case scenario” examples, that means you are “de facto” one of them, not one of us. Best explanation is the most probable one. No further evindence needed. Case closed. In my opinion, go and F your self, instead of delivering life lessons here. You have given your self way earlier. If you would have come forward and said who you are and why you are doing this, I would have respected you. Don’t struggle in the mud, you are already f/ed up. You are trying to save some people around here, maybe to convince them to stay. Good luck but things are broken beyond repair my friend.

And, one more thing, naming other companies was a smart one. Good move but not enough. Now, GTFOH.

Soporte Tecnico… podrian ser profesionales y resolver este problema de una sola vez ?

Technical Support … could be professional and solve this problem once?

come on !!!

My VPS is back up, however I cannot use FTP still… been waiting to do an upgrade all day.

thanks UC !

BTW, even my static html pages are down

is it because i told DH I quit?

anybody out here ?

I was about to post that my sites were down, but was interrupted with a phone call. So I just checked again right now, and thankfully both of them are now back up (ps52131)

Call me crazy but I’m sticking with DH. Never had a problem like this with them before. Of course that will change if things like this continue to happen on a regular basis- today was not fun! I already bookmarked many of your suggestions.

Hang in there everyone!

My VPS just came back online, but only after I rebooted it.

I’ve also been having that weird resource spike issue. Every few weeks since december I’ve been getting those “Your VPS needs help!” emails. I thought it was just me but it seems to be happening to a lot of people all at once? WTF?

Back online too, I echo Maria, never seen anything like this before but will be staying with DH, though I do have to see what my clients have to say tomorrow.

Rookie question, but how do I reboot my VPS? Is that something that is easy to do? Thanks in advance.

KT, we have those too
Never ran over 600MB while set and paying for 1200MB RAM and still getting those stupid emails

There’d better be a very detailed and HONEST explanation of what went wrong, why it took so long to fix and what Dreamhost is going to do to prevent similar problems from ever happening.

@Jennie Log in to panel.dreamhost.com On the left toolbar it should be under VPS as “Reboot Server”

Click that and it’ll give you a page where you can reboot your server.

Is rebooting the VPS server recommended? I don’t want to screw things up worse than they already are.

VPS is back – finally.

left sidebar, under mangage resources tab, reboot server few tabs down

My sites are now showing as well. Hoping they hang on. Interestingly, I read about people here with static HTML files working, and it was only after trying to access one of those pages for the first time that everything started working. Perhaps coincidental.

Just letting you know my server vps just came up!!!!

not for me , all sites including static html still down

damn you DH! all sites still down

Tuner, KT – I get those emails too. I’m going to check out JJ’s DreamHost Memory Manager as another user suggested on here. Maybe it will help.

At least I learned a few things while monitoring this post all day lol!

am I the last one here with no freaking VPS?

static html sites just came back. Iam still moving out of DH no way i will stay to watch another 24 hours of this sometimnes next week

FTP down on lansky (shared) again…

@anon

DJ Allyn: I’m a professional software developer who is well trained in server administration as well. You are here trying to claim a 100,000 record database is large!

Then quit complaining and fix it.

It is large for the resources you are setting for it. THAT is what I am trying to tell you.

You must be quite the “developer” and “server admin” if you can’t figure that out.

unless.. get this.. DreamHost is having some problem on the MySQL server causing this memory issue?

Didn’t I tell you a way you could test your theory out? Order up another SQL VPS and copy your database over to the new server and point your vbulletin site to the new database. Then turn off your original SQL server. If you are still having problems, then it isn’t the server.

Why is this so friggin’ hard for you — as a “professional” to grasp? It’s called process of elimination. First make sure it isn’t YOUR problem before you assume it is someone elses.

I am not trying to be an asshole about this — I am honestly trying to give you some things to try on your own.

@UC

It must be tough living in your world of conspiracies, where everyone else is to blame for things that happen to you. Heaven forbid you have a power outage at your house, because I’ll bet you will blame the power company for not being fast enough and for having faulty equipment.

My stuff has been back up and running fine for about an hour now. So I lost the better part of a day. If it wasn’t raining hard outside I wouldn’t have been sitting here watching people like work yourselves up in a lather over something that was completely out of yours or anyone else’s control.

You failed to notice that at the very beginning, you had a clue as to how to get your OWN website back up and at least visible. You could have simply changed your web server from Apache to Nginx and it would have at least been live again. But no, you had to sit there and complain, bitch, and moan.

Just like anon, who has all of the tools at hand to fix his own problems, but instead chooses to blame someone else.

Has any of this bitching gotten you back up and running again? That has been my point all along.

I don’t care if you move or you don’t. I don’t have a dog in this hunt. You’ve been nothing more than a salve to my boredom while I waited here on a rainy day to get my own shit up and running. Now that I have everything of mine going, I am through here.

I hope you have a better experience wherever you end up. No need to reply, I won’t be back until the next time.

All I can is this is the last straw. This is the fourth outage in the last month. I pay a few grand a year and I’m guessing that another business might appreciate it.

Thanks KT. “You have a reboot or reset for ps59368 currently pending! Please wait until this is done…”

php sites still not loading
was able to get in to the phpmyadmin

No luck with the Reboot. Oh well…it was worth a try. Thanks again.

VPS back online.

It looks like all my sites, FTP, and database driven pages are up now. Hopefully everyone else will be squared away soon too.

When I moved back to a shared server, everything was working well then I started having DNS issues. When I moved back to a new VPS, the old issues came up: .php files don’t work, .html and .txt do. I’m moving back to a shared server again since they have higher priority in the DH workflow. Goodnight.

good lord, finally!

My VPS is completely refusing HTTP connections, so it’s not fixed for everyone.

In the time it’s take them to keep telling us that is’ almost fixed I’ve already switched some of my sites over to Media Temple. I’ll have the rest done by tomorrow.

See you later DH. I will not miss your problems.

unfuckingreal. my sites are still down.

My VPS seems to be back up and running fine. Thank you Dream Host. Now that I am one of the folk who is back up, can I be a member of the conspiracy identified by @UC?

@DJ Allyn, on the issue of 100,000 being too many records to manage effectively. I agree that with most CMS and Forum software that your observation is correct. However, if designed with size in mind and properly indexed, 100,000 records is nothing for some applications. Best example that comes to mind is a table containing user number, email, password, and then the name of the db where their forum, cms, or what not is held.

Curious if you disagree with my observation.

Just ordered VPS at knownhost @ 42.50/mo for 1200MB RAM good buy DH, as soon as my freacking VPS is up and running

My VPS is back up and serving web pages. If yours is still down, reboot it or log in as an administrator user and restart your httpd!
Thanks DH for clearing a bunch of people off my host machine.

Email from Dreamhost…

Outage resolved: We believe the issue to be resolved.
(2012-01-29 19:46:02 PDT, 0 secs ago)

We’ve been restrained all day…but this is too much…don’t tell us sites are up when they’re not. You have the subdomain that bypasses the cloudflare proxy for our sites…use it.

Mine is back down again… WTFudge

Mine never went up WTF?

my sites are now up and working. this has been been the longest outage I’ve ever experienced. i would like to know what REALLy happened, was DH hacked?….

Still down. :( Ugh…and I was never back up.

im assuming this is why im getting this error

500 Sorry, no server available to handle request on apache2-cid.thimphu.dreamhost.com

Is it me or does something happend people get all butthurt and decide to split on this company, i don’t know nor care about the inner workings, as I’m sure everyone knows SHIT HAPPENS? right?? so just chill my mail actually works now.. I’m not leaving.. but i mean i can understand if this shit happens like every week every other day or at the same time on a certain yea maybe I might go explore something new. godaddy, or something of that nature, or a place over seas for what I do. but Ive been happy, they haven’t complaint to me, nor has anyone written any complaints about what I do, so i’m happy so far. but I can understand where people are coming from.

I have restarted VPS twice, restarted httpd andnothing!
50% of my static sites is still DOWN
100% php sites DOWN

all php sites still down, Moving to new host, already purchased new VPS that is it, 6 years with DH was enough

rebooting 2nd time in last 40 minutes

Thanks DH! We’re back up with more emosexual pornography than ever!

I hope to see all my new friends here, naked, in the near future.

Remember, there’s more than one gender… and I’m both of them.

Can anyone on here explain how I can restart my Apache instance? I’ve e-mailed DH, but would rather not wait for a response. Thanks again in advance.

VPS just came up, but local mySQL databases are completely missing. mysqld doesn’t even seem to be installed anymore – is anyone seeing similar problems?

I’ve learned not to rely on Dreamhost for most sites, but we do recommend them for our more budget-conscious clients. However, even that is causing problems, so I think we’ll be transitioning everyone to Servint at some point in the near future. Dreamhost is definitely an economy host and I understand that, but this sort of thing is unacceptable even with the low prices.

My VPS is back but… the SCC server is gone!?!?!? I had two software projects on it! Guys this is crazy!

this time php sites seems to be camoing back up
Great, i can begin to move them to another host

Moving out!

thisis the last time ilet DH fool me
last time it was in September

Thank goodness. I’m back up.

Sheesh 1 month of nightmares.. have started spending waaay more money with these guys and am getting hammered.. 7 years of great service and its all turned to rubbish..I hate being negative but Dreamhost VPS sucks..

Thanks Guys!!! Good Work!!!

its going to takeme a while to get out with over 30 php sites but I have no choice

Shared FTP Still Down. ETA?

I am still down! Is anyone else?

I don’t get this… in the same post you say.. //A handful of VPS servers are still being worked on at this time. If you can access your VPS,/// and then you say //All of the Dedicated Servers are online now and working!// that doesn’t make sense too me,,,,

You guys have been great for 10 years.. never had this sort of issue with you before.. I understand this stuff happens and you put effort to not let it happen again.. thats all fine.. your still fine in my book.. but please help me get back up and running asap..

You say all server are back up.. I get zero response from mine… NO MYSQL – NO WEBSITE – NO http://FTP…. not even an error page or an tempt to load the urls..

ps54241 & psmysql54279 ARE STILL DOWN……. please help in getting them back online..

Okay that was quick… Thank you DH my sites are now back up psmysql54279 — After all this time I am having like a Geezasm that all sites are now back live….. TYVM….. all my FTP is still not working and cannot be reached.. but at least the sites are up..
You still have a customer here I am just going to pretend the last 24 hours didn’t happen and that I didn’t lose a full day of business.

SITE BACK UP – MYSQL BACK UP – FTP STILL DOWN

Ps27627 still down.

It comes up, then immediately goes down.

If you use anything other than Apache, you have to manually switch back to Apache to have your site up. I think the staff got rid of Lighttpd and Nginx and the like with the recovery fix. I already sent a ticket to the support. We’ll know soon enough.

Never had an experience like this. I’m pretty sure this was the last time using dreamhost… I’m moving to bluehost.

January 30th, 2012 at 12:50 amStéphane Dion Says:

“An Apache restart is fixing most issues, which you can do manually”

/etc/init.d/httpd2: line 16: ulimit: open files: cannot modify limit: Invalid argument
/bin/ls: cannot access /dh/apache2: No such file or directory


Stopping httpd-ps15841 webserver…done.
Starting httpd-ps15841 webserver…grep: /proc/6175/cmdline: No such file or directory
starting…rm: cannot remove `/dh/apache2/httpd-ps15841/logs/__db.session_cache': Permission denied

this lasted for 2 hours, every 5 minutes. this seems like a very scientific method to bring things up!

Oh well. At least I am not receiving the DAILY reboot emails because of memory exceed limit with my VERY LOW traffic websites! and this after I have upgrade my memory limit from 300MB to 400MB.

stop screwing us! you are already well paid!!

Try to really up your memory, say to 2GB, for a couple days and see if it solves the problem. Than you go down gradually. You really can’t tell the effective RAM usage or need by looking at the graphs of the panel: they’re just averages and your server can be brought down by very short peaks that don’t show on the graphs.

Oh well. At least I am not receiving the DAILY reboot emails because of memory exceed limit with my VERY LOW traffic websites! and this after I have upgrade my memory limit from 300MB to 400MB.

Oh thank god! I’m not the only one? Is anyone else having this?

All sites, all email, all everything — still down. VPS restarted, no effect. Support ticket sent, but will probably be ignored in the flood. I can’t even get on long enough to move my crap elsewhere.

Seventeen-year relationship with this company, and this is how we’re treated…? Looks like today will be spent procuring a hosting company that actually gives a flying rat’s ass about its customers.

Organize backups elsewhere, on your machine or (better) by other hosts, every day or every minute depending on your level of activity. If your main host go down for good, you’re up and running in just the time it takes to move the DNS. Nobody out there is going to give you a perfect secure server for a few dozen dollars a month.

fuck that shit, 24 hours over

I GOT AN EMAIL SAYING FTP ON MY ACCOUNT IS FINE BUT I STILL CANNOT ACCESS FTP. I’M IN THE UK AND LOST AN ENTIRE DAY OF BUSINESS BECAUSE OF THESE OUTAGES. I UNDERSTAND THAT THINGS HAPPEN BUT TO HAVE MY SITES GOING DOWN AND NOT BEING ABLE TO ACCESS FTP FOR OVER 24 HOURS IS COMPLETELY UNACCEPTABLE. IF I HAVE ANOTHER DAY WHERE I CAN’T ADD NEW STOCK TO MY SITE THIS WILL BE A WEEK OF FINANCIAL RUIN FOR ME!

IS DREAMHOST COMPENSATING LONGTERM CUSTOMERS SUCH AS MYSELF FOR THIS ISSUE? LIKE A OR OF FREE HOSTING OR SOMETHING? ANYTHING? WE’RE AT YOUR MERCY HERE AND THINGS AREN’T FIXED! THIS IS TWO WEEKS IN A ROW YOU’VE HAD MAJOR OUTAGES, ARE YOU GOING TO GET THIS FIXED?

I JUST KEEP GETTING MESSAGES LIKE “NO SERVER AVAILABLE TO HANDLE REQUEST” “CONNECTION CLOSED WITHOUT INDICATION” WHEN I TRY TO CONNECT TO FTP.

I’VE SENT MULTIPLE SUPPORT REQUESTS AND AM NOT GETTING ANY HELP! PLEASE PLEASE PLEASE HELP ME AS SOON AS POSSIBLE!!! I’M SERIOUS, THIS IS COSTING ME MY BUSINESS AT THE MOMENT AND WITH SO FEW DAYS UNTIL THE END OF THE MONTH IF I CAN’T ADD STOCK TO MY SITE I WILL NOT MAKE MONEY THIS WEEK! THIS IS SO SERIOUS!!!

FTP DOWN ON STAMPEDERS, FTP STILL NOT WORKING!

Robert, you are not the only one losing business!

Dreamhost, I know you USUALLY have wonderful support but lately there have been quite a few things going wrong. Users that are effected should definitely get some sort of compensation.

I’ve just gone down again.

I was working for about 10 minutes before it happened.

I’ve sent a ticket to support, but suspect they’re buried still.

Interesting that today’s outage happened at about the same time as the one yesterday….

WAIT! STAMPEDERS IS BACK!!! SLOW, BUT BACK!!!

I know I’m just one customer and I’m sure Dreamhost doesn’t care, but this downtime cost me a great deal both financially and in terms of the integrity of my website and I doubt I will ever really recover.

Regardless of why or how it happened, a catastrophe like this, made worse by the slow recovery time and non-existent customer service response is really a deal-breaker for me.

I can’t speak for anyone else but personally I have my future staked on my website and to have it all wrecked like this is just devastating.

Without some sort of compensation I am going elsewhere. Not an idle threat made that will be forgotten tomorrow, I will quit Dreamhost if they don’t take steps to compensate customers for this disaster of THEIR making.

Yes, I know, Dreamhost doesn’t give a shit about me and my account, and they won’t care one bit when I leave. But what does that say about them as a company?

Robert: try to change (or just re-enter and save) your ftp password in the panel (Users –> Manager users). It just worked for me with 2 ftp sites that were stuck.

I had 150 domains offline for almost 20 hours. I have to compensate my customers, how are you going to compensate us ?

FTP access still down for my VPS.. tried resetting password in the panel (Users –> Manager users), waiting 10 minutes, and rebooting server, it doesn’t work. May as well have the entire site down still, without FTP access its useless.

GIVEN I’M DOWN AGAIN (FOURTH TIME IN FOUR DAYS) I’M DECIDING WETHER TO LISTEN TO:

I fell in to a burning ring of fire
I went down,down,down
and the flames went higher.
And it burns,burns,burns
the ring of fire
the ring of fire.

(johnny cash’s ring of fire)

OR Aerosmith’s ‘i’m down’

You telling lies thinking I can’t see
You don’t cry ‘cos you’re laughing at me
I’m down (I’m really down)
I’m down (down on the ground)
I’m down (I’m really down)
How can you laugh when you know I’m down
(How can you laugh) when you know I’m down

¬¬

January 30th, 2012 at 5:30 amNow REALLY Frustrated Says:

Simon says they were messed up because of the automatic updates to their operating system. Are you kidding me? What kind of tech shop automatically updates production systems without any testing? COMPLETE AMATEURS!

Automatic updates in a system that HAS TO BE STABLE? Guys, back to earth! Nobody does this kind of procedure over a 24×7 system!

This is very sad. Disappointing my readers. I was at wordpress.com for 5 years and no downtime. I really regret to have moved here. I just have to go back to wordpress.com. I find what is happening here very saddening. Down almost the whole weekend…… what is this …..

i have been here for onle 7 days, this happenning is no encouraging me to host here.

Correctly written: Correction:

I have been here for only 7 days, this happenning is no encouraging me to continue hosting here.

Does anyone know if I do the manual Apache server reboot in the admin, will my sites be down for a long time?

An Apache restart takes less than a minute. Just FYI.

Peace,
Gene

UP ?

DOWN AGAIN

SUPPORT ? NO SUPPORT, MY LAST TICKET 4 hours 22 mins NO RESPONSE

YERTERDAY ALL DAY DOWN

THIS IS AN INSULT TO ALL

THIS IS THE REALITY OF DREMHOST

DREAMHOST IS NO MORE DREAMHOST. IT’S NIGHTMAREHOST INSTEAD.

Opened a ticket 30 minutes ago, but by the looks of things, I don’t expect a response soon.

I tried to reboot http2, but I get the following error:

/etc/init.d/httpd2: line 16: ulimit: open files: cannot modify limit: Invalid argument
/bin/ls: cannot access /dh/apache2: No such file or directory

WTF?

Apache server restart didnt fix the FTP issue. Our sites are up, but I get a 999961 error in my FTP program when I try to connect.

We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers.

There you have it: DH has told us the issue and as expected, it was something unexpected from the Open Source crowd. (This can also happen with closed source OSes, but seems less likely to do so of late. Then again, I don’t want to be hosted on Windows. Been there, done that, fifteen years ago.)

It would be really great if DH extended some freebies to us all, but it’s important to remember that DH are also victims in this case. I’m sure having a sudden panic and 24 hours of massive workload really did their heart rates a lot of favors.

I have to say I’m really disappointed in many of the people complaining, threatening, whining, etc. in this page of comments. Your behavior did not help the situation, achieved nothing, and in general exemplifies what I find distasteful about humanity.

I have been hosting with DreamHost for 10 years and this is without doubt the worst incident I have experienced. However, it was just one very bad weekend which cost time and money compared with 10 years of excellent service. I’m sure most hosting companies (and most businesses) have a ‘lost weekend'; but they learn from it as I am sure DreamHost has done. I have been too long with DreamHost to consider moving because of one bad experience.

VPS FTP still down. 15 hours later. So sad.

@Random Emosexual – You’re a douche.

Yesterday : Trying to put up the new website for my gf’s company launch. Couldn’t put it up. BIG LET DOWN.
Today : Have to transfer files via ftp legally comply with a provider here : Can’t log into my ftp.

I am so disapointed. I won’t renew my subscription with Dreamhost unless something comes up.

@Charles. Did you change your FTP password? All FTP passwords were reset last week after the security incident. My FTP is working fine.

Many FTPs are still down.

“Site Temporarily Unavailable – We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf””

I’m a new client for a week. Just tried to transfer some sites today, but it didn’t work: FTP down, 1 site I created as a test is also down… I’m coming from another company and I cancelled my subscription because they where a little bit to expensive for me. Dreamhost looked like a better choice, but I lost so many hours today… so Dreamhost is already more expensive to me then my last host :-(. I read there are a lot of problems? 1 other problem in the coming month and I’m out of Dreamhost, that’s for sure!

Now the panel is down. Can this get any worse?

…and no update since last night. Really?

I would love to restart apache on my server, but even the panel is down now….
https://panel.dreamhost.com/index.cgi <- broken

oh my living jesus, moving here 7 days ago has caused me disaster. i wish i had no come to dreamhost. nothing is happening. i am stuck and i have readers to satisfy. moving out as quickly as i can.

FYI,

The Dreamhost web panel is down at this moment.
http://panel.dreamhost.com
https://panel.dreamhost.com

Is this issue related?

Panel is down, all websites down for hours now. wtf come on

Panel is down? Perhaps that means DH staff is awake and kicking the tires (8:00am Pacific… sounds about right).

Site is still down, cannot connect to web panel to open ticket either. Please don’t post that issues have been resolved when sites are still down after 24 hours.

OK, THIS IS TURNING INTO NIGHTMAREHOST. FTP STARTED WORKING BUT NOW MY WEB SITES ARE DOWN! WTF IS GOING ON???!!!

Nice. I can’t even get in through VPS now. I was going to d/l my most critical sites with spc and upload to a temp server, but now I can’t even do that. I’m checking news.google.com to make sure Los Angeles didn’t fall into the ocean.

I can’t get Dreamhost.com to load or the panel either so I can submit a new support ticket! Anyone else having this problem too?

Still down and panel still down. Total BS.

great – everything was back to normal this morning, up until now – everything is down again. what a fucking useless host.

Now I know why I received a credit for the whole month of Jan when I complained about all of the down time in the last few weeks…. I’m to expect more downtime.

Can’t get to the panel at all as well as all of my websites.
Any updates on this DH?

i cant acces panel dreamhost or web mail service waht happend i need your help????

Wow, what a shitshow, Dreamhost. Tried accessing my websites this morning to see they’re all down, and I come here and what do I find? Yet another 900+ comment issue. Panel is down so I can’t even file a ticket. Usually I’m on your side, saying “you know, shit happens sometimes” but damn if it isn’t hard to say that after the recent snafus.

Why is THIS page still up when everything else is down?

from italy
+

All DOWN…… AGAIN AGAIN AGAIN ……

Panel.dreamhost.com not work, panel down, all site down all mail down

I have found the service here VERY substandard. I’m going to be transferring the all hosting to SPIRAL HOSTING. They haven’t let me down in the past and they keep customers in the loop.

http://www.spiralhosting.com

POP, webmail and panel all down. FIX IT.

Everything is broken, websites are down, my VPS is down. What is going on?

SITES DOWN WTF!!!!!!!!!!!!!

What is going on? I cannot access the DreamHost Panel and all of my websites are down!?!

Yeah. in the same boat here. Found this problem yesterday. Ignored it until 7pm. Reported it. But still down 13 hours later, along with the panel, wiki and SSH, so I’m locked out completely. Faith is getting harder in the face of these recurring downtime issues. I’ll be taking plenty of hell for this from my superiors. Please, please, please, can we have things fixed?

ALL DOWN AGAIN

SUPPORT ? NO SUPPORT, MY LAST TICKET 6 hours, NO RESPONSE, NO CHAT

YERTERDAY ALL DAY DOWN

THIS IS AN INSULT TO ALL, THIS IS THE REALITY OF DREAMHOST

The panel is down, but my sites are up and the ftp works fine (on a vps that was down yesterday).

Opening an account with BlueHost and uploading my files ASAP. Of course, since my DNS and registration is hosted at DH, it won’t do me any good until I can get the damned panel open.

All down, all down, all down again.

WTF !

As yesterday down again!!!! I am still crying about all my lost yesterday wtf are you doing ruin our biz? What about compensation for all this ? member for 6 y now…but this is “no words”

Site down and panel down so can’t submit support ticket

Ftp, panel, svn are down :(
Please answer…

After 24 hours of down time yesterday…

I am speechless.

Site and email is down. Control Panel is down.
No way to submit a trouble ticket.

FUCK THIS SHIT

This is beyond FUBAR.

Uh, pardon me, let us not start `bitching` again. Some retards from yesterday will start singing now. `Don’t bitch, this is normal, we have seen worse`…

Everything is down, cant get on panel to submit support ticket.

FTP, www, cpanel etc…

FIX THIS SHIT NOW!

THIS IS NEW:

As of 8am PST, we are seeing issues connecting to our multiple datacenters. We are getting everyone from the admin, network, data center and support teams on the case to get this issue resolved as soon as possible. We will update regularly with any info that we have.

NOW INSTEAD OF ASAP WOULD BE PREFERABLE. NO THANK YOU.

Dreamhost is under attack :( , I’m worried about my data now.

I am suspecting now , someone is attacking them brutally.

Their main site (dreamhost and panel) was on a different hosting. Why is it down now?

They have been lying to us since the beginning, there is something else going on here.

Everything is down, can’t get on panel to submit support ticket.

FTP, www, cpanel etc…

yup its still down and out. ftp, website, email – everything. and yep, cant even send in a ticket or contact support, no nothin nowhere!

This next one goes out to my peeps…

CAN’T SEND E-MAILS.
CAN’T ACCESS WEBMAILS.
CAN’T ACCESS DREAMHOST.COM
CAN’T ACCESS PANEL.
CAN’T GET IN TOUCH WITH ANYONE.
NO ANSWERS.
NO SUPPORT.

BOOYAH

This is a joke.

I think is the worst outage in Dreamhost history.

Can we start listing out Dreamhost alternatives?

Hostgator
MediaTemple

Where else?

Slicehost

MORE OPTIONS PLEASE

OK, I’m going to lunch. If the internet doesn’t comes back on, let me know so I can just not come back into the office…

Sweet! home page and VPS is back up! I’m downloading my sites NOW. Bluehost.com, here I come!

It is back up for me now…

Actually after this outage I believe Dreamhost is a much better option of what it was before. I think they are learning a lot, and they will turn this problem into an opportunity. I’m sticking with Dreamhost.

Also is clear that many of the people posting “Options” are just people from other hosting companies.

Loosing a client over these issues. Now sites are back up but no shell or ftp access preventing updates or access. Clients are livid and client trust over quality of service has been crushed. Very disappointed.
I have three questions.

1. Don’t you test before you impact all your customer?
2. Don’t you test before you impact all your customers?
3. Don’t you test before you impact all your customers?

To be honest I’m pissed off that there hasn’t been an update to this issue in 12 hours… 12 HOURS!

Is it really “clear” that the people posting other “options” are from other hosting companies? Really?

compensation? payback to users?

one of my sites still down. Bye bye DH. I’m fed up, enough is enough. Congrats, you’ve just lost another customer.

Well I found a SOLUTION TO THESE DREAMHOST PROBLEMS, by switching my acct. a few weeks ago. THANK GOD.

The solution was using reseller accounts to host my own sites. The purpose of the reseller acct. is intended for using it for clients, which I’m expanding to now… BUT IT’S WORKING EXCELLENT for my sites now also.

No more rising ram rates on VPS. Reseller acct. is only costing me $25/month for many sites.

CLICK ON MY NAME, and check out the resller acct. where I’m a happy customer.

Today I learned:

People on the internet are insane/dense and make all sorts of statements that do not correspond to reality.

I’m honest about myself… I’m just here to find partners for genderless sex.

you fucked my biz what about compensation? wtf tell us

@Random Emosexual fuck off

@Random Emosexual fuck that off

reading to many books @Random Emosexual but still fuck off

Kiss me boys!

Touch me girls!

I’m sensing a class action law suite in the making.

Dream Host you will be hearing from my lawyer. The amount you’ll owe is not going to be pretty.

Now I can’t even send support tickets.

Dream Host you will be hearing from my lawyer.

99.9999% of threats of this nature on the internet are 100% bogus.

But if your lawyer’s cute, you can sue me. In my rectum.

January 30th, 2012 at 10:17 pmDreamhost!!!!! Says:

I understand you’re working on it, but delete this “random emosexual”. Someone should be following the comments, even if they are angry to keep this guy off of them. Let’s be professional. Clean it up and delete posts like his.

i switched over to E2 Hosting. You might want to check them out! :D DH, you suck!

https://bitly.com/cheaphosts

My FTP access is still down…

Client since 2003.
It bogles the mind to learn that “pros” didn’t know autoupdates should be off from the very first install.
No way could you trust productions servers to autoupdating!
I’m sorry, but it really makes one lose faith in you guys…

This is unbeleivable for such a big company like this.

yup everything is broke ..

Have you been looking into sudden permissions issues for shell account too?

I am having some weird stuff happen with one account, where no changes are respected, and despite the appearance of proper permissions, errors that say changes can’t be made… I also am having problems viewing contents of some directories when connected as shell user.

I already submitted a support ticket but wanted to talk about it generally too.

Hi guys! I hate to complain, but I’m starting to lose customers, and I’ve been spending WAY too much time on the phone handling complaints from those customers I have left. I understand that problems happen, and that things like this will happen from time to time, but this is really starting to affect my poor, tiny, starving business…

My VPS is up and running (yay!), but I’m getting complaints that CMS isn’t working. (I’ve been trying to wean my customers off CushyCMS for a few years now, but I still have several holdouts that refuse to change.) Anyone have any ideas if this is related to the current spate of problems?

@April .. yes same problem here permissions are messed up

FTP is STILL DOWN. 23+ hours waiting for a response to my Red Alert ticket. Am really, really, really fed up. :(

January 31st, 2012 at 11:32 amNow REALLY Frustrated Says:

Lovely…support says to get your VPS back in shape, contact support. Except the panel is down and I can’t seem to contact support. WTF?

yes panel down

but sftp is still up
quick get your carp

Panel is down for me.
Appreciate the update on the cause and change in policy, but more updated, public problem acknowledgment and ETAs would be appreciated.

panel back up ..

January 31st, 2012 at 12:43 pmwow you guys are filled with fuck Says:

so some idiot runs ‘apt-get update’ on your live servers and you’re down for three days? this is the most awesome thing.

it’s time for us to stop whining here, and start whining to the press. dreamhost has moved beyond simple incompetence and potential customers need to know this in advance.

mysql is down.

hit the reset button ..back to normal

Back to “normal” here too… today I’m a woman, tomorrow a man, and the day after that, genderless!

February 1st, 2012 at 11:29 amRANDOM EMOSEXUAL Says:

I’M NOT FUNNY,
I’M NOT CLEVER,
I NEED ATTENTION,
I NEED HELP,
MY PARENTS TOUCHED ME,
THEY DIVORCED BECAUSE OF ME,
I’M NOT LOVED BY ANYONE,
PEOPLE MADE FUN OF ME WHEN I WAS YOUNG AND I TAKE IT OUT ON THE INTERNET,
I’M NOT ORIGINAL,
TROLLING IS OLD AND BORING,
MY ONLY OUTLET IS TO MAKE JOKES ON THE COMPLAINTS SECTION OF A RANDOM WEBHOSTING BOARD…

February 1st, 2012 at 11:35 amrandom emosexual Says:

everyone can see my pain by the way i troll…

and we do…we can see you’re pain. you’re hurting. you need help inside. or let the pain out with a knife to your veins, just let the world forget about you, because you’re not wanted, no one wants you, your body, your mind, your opinion, no one wants you.

NO ONE WANTS YOU!

Looks like random homosexual is back on his “daily shift” again. Actually no one has ever heard that he has a web site except the porn site he lists as his. He does not because he is undercover DH moderator, his job is to calm down the people who are furious toward DH. So people dont even bother what he is saying. He is a borderline retarted with an full scale IQ around 70s, working part-time for DH to make living, and guess what, we are not supposed to know but many of us know already know it.

It came back but I lost absolutely everything on the server, files and DB. Good thing I had a backup

Switched my vps for Iweb it cost a bit more but if you take 4gigs of ram it’s half dreamhost price 100$ instead of 200$. Plus CPU power is way better ( Core 2 Duo )

And you have wayyyyy more control over your vps.

I had a hosted services with Dreamhost few year ago, I’ve experienced some pretty bad slowdown. I’ve asked customer service and they said everything was normal. So i’ve switched. I was hoping more from the vps cause I pay more money . I was wrong. A bad hosting compagny is a bad hosting compagny.

In the end this crash was a good thing. Cause I’ve discovered a way more reliable, fast, cheap and professional hosting solution.

He does not because he is undercover DH moderator, his job is to calm down the people who are furious toward DH.

Uh… I think Dreamhost would hire just about anyone other than me. The whole point is that I’m an unstable troll and the rest of you make me look like Abe Lincoln.

“Oh look, my website is down! I think I’ll log on and:

(a) Threaten my webhost
(b) Insult my webhost
(c) Insinuate I’ll sue my webhost
(d) Compare my webhost to others
(e) Talk loudly about leaving
(f) Wail, gnash teeth, scream obscenities”

What part of that sounds rational to you? Yet it’s what 95% of the people here are doing.

All I’m doing is trolling for genderless sex. I was told this was the happeningest place on the internet for that.

Random homo, what you have been consistently doing :

1.Spending time here, day and nite – for no reason?
2.Selecting people who are frustrated for a good reason and trying to make them look like they are “irrational”.
3.Making it look like you are “anyone”, trying to distract people by this way, your meaningless nickname has successfully attracted some attention here, despite your contentless comments.
4.And thinking that you look like Abe Lincoln? Really? :)

How could DH find a better impaired than someone like you who has nothing else to do but is willing to spend all days and nights here on webhosting board? (depends on how much they are paying you, probably not much)

Chances are high that you have a good reason for what you are doing,whoever you are is of no importance, because if you do not have DH connection, that qualifies you for a “real” imbecil with a FSIQ below 60. If you are, which chances are high, that qualifies you being a “real” loser who is really so desperate to make money from a disgusting job.

So either way, you are a loser, you know it and I know it :) Good luck to you…

@UC:

You are still here? I would have thought you’d be in greener pastures by now.

I see you are still accusing people of being “undercover moderators” for DH. Apparently it doesn’t pay much, because I still get charged $80 a month here.

All of that anger directed to a company that had a serious issue going. Now that it is over, I think they did a pretty good job, considering.

@sesame “I’m paying like $80 – 90 for daily traffic of around 12k.” – where did you buy this advertising?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Maintenance scheduled for MySQL server ‘madal’ on Feb 5th @ 9pm PST (RESOLVED)
6:31 pm

Maintenance scheduled for MySQL server ‘madal’ on Feb 5th @ 9pm PST (RESOLVED)

Posted (January 26th, 2012 at 6:31 pm PST) by Dan J

We’ve scheduled a window for maintenance on the MySQL server ‘madal’. The maintenance will begin Sunday, Feb 5th at 9pm PST. The window should be limited to 30 minutes or so during this time and will only affect MySQL related services. It will only affect site availability if the site is using a database that is hosted on this server. We will update this post shortly before the maintenance window starts and update it when it’s complete. If you have any questions regarding this, please don’t hesitate to contact support to let us know.

Update Feb 5th, 9:24pm PST: The maintenance has completed successfully and the server was only down for about 15 minutes. We appreciate your patience and hope this improves the server’s performance. If you run into any problems, please contact support and let us know! -Seohee

This entry was posted on Thursday, January 26th, 2012 at 6:31 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

16 Responses to “Maintenance scheduled for MySQL server ‘madal’ on Feb 5th @ 9pm PST (RESOLVED)”

my website is downnnn help please hollstein@gmail.com help!!!!! chat live not working… bad service

WTF!!! AGAIN!!!!

TOLD U BEFORE U WANKERS, NOW YOU BETTER REFUND ME BEFORE WE HAVE MORE PROBLEMS!!!!!

LOSING BUSINESS HERE, GONNA MAKE U TWATS COVER THE COSTS YOU JEW FKS U!

fomalhaut is down

goldfish:deaner is down too.

– got 10 site down :(

What is happening?

V.

No it’s up & running

(now)

psmysql62687:psmysql62687 is apparently down.

my dedicated server is up, but the sites are not showing. They use mysql but is this also effects dedicated servers or was it only for virtual and shared hosting? Please be more specific and clear about maintenance jobs.

@Dude: This is a maintenance that is not scheduled to happen for another week, and is very specific to one mysql server. If you are having issues at this time, it is not related. Please check the other status posts or contact support for further assistance.

Should we have a moment of silence for all the people who killed themselves in drama-gasms over the outages?

again https://panel.dreamhost.com/ is down. whats going on

Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

then where the hell to post comments or tell to DH that they again sucked.

Again, webmail slow as molasses. Made the jump to another host today.

I use mySQL to keep track of my countless sexual partners, and the database is going down like a Thai ladyboy.

Can’t you see I’m losing customers? Also all those emails about Viagra… I always flirt back!

My server is down, rebooted with no luck. ps75428

second time this afternoon that all sites are down, losing money as we speak and chat has been off permanently for days

very angry

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Connectivity issues to a small subset of web servers (RESOLVED)
2:40 pm

Connectivity issues to a small subset of web servers (RESOLVED)

Posted (January 26th, 2012 at 2:40 pm PST) by andrea

Currently, we’re experiencing connectivity issues to a small subset of web servers in one of our datacenters. Our Datacenter Operations team is investigating the problem, and we will post details when we know more.

The affected servers are:
pigwidgeon eskimos homie-vserver90 alouettes bluebombers tigercats argonauts stampeders roughriders olympiacos herediano d-fenders bluejays homie-vserver89 homie-vserver91 homie-vserver92 homie-vserver93 homie-vserver94 bandar fever revolution saprissa fire crew rapids earthquakes shock liberty monarchs islanders blackhawks predators lightning thrashers flames dynamo

You can tell if you’re affected by checking ‘Your Web Server’ under Account Status on the upper right corner of your DreamHost panel. If you’re on a VPS and are experiencing downtime, please check the upcoming update for the list of VPS guests affected.
Please note, this does not involve any other machines, or any other services such as mail or mysql. – Andrea

Update Jan 25th, 3:05pm PST: We just received confirmation that the problem with the above servers is now resolved. We had a temporary issue with the switch servicing those machines: it wasn’t able to reach the internet. It is now fixed. We apologize for the inconvenience this has caused. -Andrea

Update Jan 25th, 2:53pm PST:  The following VPS guests were affected:
ps14427 ps14509 ps14528 ps14547 ps14553 ps14558 ps14567 ps14571 ps14585 ps14602 ps14609 ps14651 ps14851 ps15006 ps15012 ps15019 ps15021 ps15075 ps15119 ps15133 ps15230 ps17503 ps17591 ps17641 ps19904 ps25570 ps28532 ps28545 ps29087 ps29365 ps29599 ps29650 ps30138 ps30946 ps34008 ps38072 ps42499 ps44190 ps44200 ps46464 ps46471 ps46804 ps47524 ps55740 ps56582 ps56584 ps56588
ps14497 ps14514 ps14539 ps14574 ps14608 ps14615 ps14645 ps14660 ps14683 ps14694 ps14703 ps14706 ps14707 ps14712 ps14871 ps14872 ps14884 ps14929 ps14931 ps14932 ps15088 ps15202 ps15207 ps15293 ps15304 ps15642 ps17018 ps17080 ps18332 ps18335 ps18388 ps25583 ps26790 ps27670 ps27778 ps28144 ps30016 ps30421 ps30798 ps33963 ps41362 ps43803 ps54721 ps55793 ps56467 ps64525 ps72227
ps14434 ps14522 ps14529 ps14535 ps14559 ps14568 ps14618 ps14642 ps14648 ps14662 ps14680 ps14873 ps14875 ps14882 ps14885 ps14922 ps14924 ps14952 ps14963 ps15134 ps15326 ps15485 ps15568 ps15586 ps15724 ps15738 ps15760 ps15915 ps22048 ps22447 ps23229 ps23235 ps23402 ps23447 ps23793 ps23840 ps23876 ps25514 ps25683 ps26683 ps26875 ps28023 ps33457 ps34023 ps34039 ps35073 ps37520 ps38913 ps41346 ps43983 ps47598 ps48429 ps53452 ps53567 ps53568 ps55800 ps56666 ps62748
ps14435 ps14500 ps14507 ps14513 ps14546 ps14562 ps14594 ps14601 ps14607 ps14614 ps14676 ps14888 ps14889 ps14953 ps14994 ps15131 ps15151 ps15272 ps15273 ps15303 ps15709 ps17650 ps17700 ps17744 ps17894 ps17906 ps17940 ps17955 ps18260 ps18302 ps18315 ps18316 ps18499 ps18500 ps18543 ps18550 ps18607 ps19771 ps20352 ps33482 ps35476 ps41987 ps42560 ps43020
ps14430 ps14501 ps14511 ps14523 ps14536 ps14549 ps14560 ps14597 ps14612 ps14619 ps14626 ps14637 ps14654 ps14677 ps14682 ps14730 ps14754 ps14774 ps14856 ps14863 ps14877 ps14918 ps14919 ps14920 ps14951 ps14972 ps15003 ps15022 ps15113 ps15165 ps15233 ps15619 ps15739 ps16196 ps16404 ps16444 ps17623 ps18187 ps18347 ps18609 ps18774 ps19460 ps28752 ps31018 ps32813 ps34798 ps35060 ps35640 ps38706 ps38727 ps39251 ps42285 ps43425 ps43862 ps43911 ps47534 ps48498 ps48515 ps49733 ps54047 ps54331 ps55499 ps55610 ps56309 ps56914
ps12708 ps14431 ps14518 ps14590 ps14600 ps14606 ps14689 ps14691 ps14692 ps14693 ps14695 ps14827 ps14868 ps14893 ps14902 ps14926 ps14947 ps15091 ps17601 ps17604 ps18340 ps18508 ps18510 ps18511 ps18522 ps18524 ps18527 ps18534 ps18575 ps22383 ps22666 ps23205 ps23228 ps23250 ps23618 ps24089 ps24308 ps24344 ps24632 ps25297 ps26045 ps26530 ps27976 ps33548 ps34059 ps34203 ps34692 ps34801 ps35467 ps35997 ps36084 ps36150 ps47538 ps48184 ps64630 ps71704 ps72772 ps73396

This entry was posted on Thursday, January 26th, 2012 at 2:40 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

9 Responses to “Connectivity issues to a small subset of web servers (RESOLVED)”

I am so disappointed!
I’ve been with DH since 2006 and this month has been the worst ever.

My sites have been down at least 4 days this month!

horrible!

Hurry up. Just posted big article and your killing my traffic.

I think its about time that everyone switch to http://www.rackspace.com

I’m done with this company.

All hosts have problems.

It’s how they handle the problems that makes one host better than another.

I’ve used a lot of hosts in the last 19 or so years, and nobody has been ON IT better than DH.

When times are good, nobody takes notice.

OTOH, we moved our email to a different dedicated host. :)

I cannot access my FTP for the second day. The issue is obviously not resolved and I’m not on any of the affected servers. You may need to look into this more.

thanks for a quick resolution yesterday!

I cannot access shell, is this related? My server wasn’t on the list but the symptoms seem the same.

My server is belgrade, which is not on the list, but I haven’t been able to access it for 3 days.

You know what I look for in a host?

Well… I guess the answer is “pornography.”

But that’s another story I’m sure.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘torr’ moved to new hardware and restoring from backups (RESOLVED)
8:39 pm

Shared web server ‘torr’ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 23rd, 2012 at 8:39 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘torr’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server torr, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Jan 25th, 9:58pm PST: The restore process has finished! If your site is missing any recent content then please contact support and we can attempt to restore the data from the old hardware. Thanks! -Dan J

Update Jan 25th, 1:28pm PST: The restore process is continuing to run smoothly, and all services remain online and are being served from our backup server. Looks like we’re getting close to the end of the restore. We will continue posting additional updates as the restore progresses. Please contact support with any questions you have! -Seohee

Update Jan 25th, 10:29am PST: The restore process is running smoothly, and all services remain online and are being served from our backup server. We will continue posting additional updates as the restore progresses. Please contact support with any questions you have! – Patrick

Update Jan 25th, 7:35am PST: The restore process is running smoothly, and all services remain online and are being served from our backup server. We will continue posting additional updates as the restore progresses. Please contact support with any questions you have! – Chalise

Update Jan 25th, 5:27am PST: The restore process continues to run smoothly, and all services are online and being served from our backup server. We will post additional updates as the restore progresses. Please contact support with any questions you have! – Chalise

Update Jan 24th, 10:43pm PST: The restore process appears to be functioning properly. All services are online and being served from our backup server. We will continue to post updates with the progress of the restore and if we encounter any issues with the process. Please contact support with any questions you have! -Dan J

Update Jan 24th, 4:43pm PST: Please note the change to the post content above, as this server is now being restored to new hardware following a failed file-system check. The new hardware is in place and all services are online and being served from our backup server. We will continue to post updates with the progress of the restore and if we encounter any issues with the process. Please contact support with any questions you have! -Seohee

Update Jan 24th, 2:56pm PST: Please note the change to the post content above, as this server is now being restored to new hardware following a failed file-system check. The new hardware is in place and being served from our backup server. There are currently some Apache services down, which our admins are working on now. We will continue to post updates, so please check back soon. Please contact support with any questions you have! -Seohee

Update Jan 24th, 11:27am PST: Please note the change to the post content above, as this server is now being restored to new hardware following a failed file-system check. The new hardware is in place and all services are online and being served from our backup server. We will continue to post updates with the progress of the restore and if we encounter any issues with the process. Please contact support with any questions you have!

Update Jan 24th, 7:00am PST: We are having a few issues with one of the apache services on this server, however, this should not be affecting the restore process and the majority of sites on this server are up and running. We have an admin continuing to monitor it and hope to bring all services back online as quickly as possible. Further updates to come as we have them. – Chalise

Update Jan 24th, 3:55am PST: The restore is continuing, however, it looks like Apaches on this server are having a few issues running at the moment, which is causing sites to be unable to load. Our admin is currently looking into this, and we will continue updating as we have further news. – Chalise

Update Jan 23rd, 10:51pm PST: The new hardware is now online. Sites should be loading data from our backup servers so all site should be online. If you find any problems then please contact support. We will post more details about the restore here.  -Dan J

Update Jan 23rd, 9:11pm PST: Our admin team are currently rsyncing some files to the new hardware before they bring it online. Once that’s complete the admins will try to bring the new hardware online. It shouldn’t take much longer. The new hardware will then be using data from our backup machines. -Dan J

This entry was posted on Monday, January 23rd, 2012 at 8:39 pm and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

38 Responses to “Shared web server ‘torr’ moved to new hardware and restoring from backups (RESOLVED)”

That explains FTP 550 Read-only file system?

This is completely unacceptable. We are trying to run a business here and you take down our site with no warning??? And, now, 9 hours later, we are still down. This is the worst service imaginable and I need to SPEAK with someone immediately to get this resolved!

Site is down for more than 12 hours! Im loosing money! Its unacceptable!

Online chat is offline since yesterday also! Its unacceptable!

Support ticket not answered until now! Its unacceptable!

Four hours later and we’re STILL down! That’s a great 100% uptime guarantee. And although they say that “the majority of sites on this server are up and running” we know of at least one other customer who is still down.

Make that three other customers. I think they are lying to us! How about a class action law suit?

I double the questions Rance asked. If we could get some feedback that would make this a little more bearable

this says that the server “torr” is having issues. my machines says it’s “keid” so why am I having issues as well?

All my sites are also down and I am NOT on torr. No replies from customer support.

I have one website down, while the others are up, and I don’t have torr either… don’t know what’s happening. No replies from customer support too :(

It came back to life exactly at the same time I posted the previous comment.

All of our websites are still down for over 15 hours and fTP and we are NOT on torr server! No update since 7AM?? We are on East coast and it’s 11AM!!! Unacceptable! All chats are turned off for months.

Yep, same here. All sites down, ****not just down – all website files actually MISSING from within the FTP directories**** and we are NOT on torr.

No communication from customer support so I have to resort to posting here. I know Dreamhost will never respond / look at these but at least other customers can see that other people are having the same problems.

All my sites are down as of 11:30am EST. whats up?

Well, folks, at 11:50am, we came back up again. Very nice 14 hour outage.

It is now January 24, 2012 12:09 pm EST and we are still down. WHY? Will some one please respond?

We are also STILL DOWN with no replies from support. Someone from Dreamhost please update this status. Many of us who are down are not even on torr including me.

We are also not on the torr. We are losing so much money.

I seriously feel for you Julia. We are in the same boat. Our sites have been down (on and off – mostly off) since Sunday morning. We haven’t received any communication from Dreamhost. The fact that nothing is even posted on their status about the downtime (seeing as we’re not on torr) is just unacceptable.

FUCKING SHIT!!!!!!!!!!!!!!!! ALL THE TIME!!! SORT IT OUT FFS!!!!!

U GOT 1 MORE CHANCE THEN IM MOVING SERVERS

THIS IS UR FINAL WARNING, REMEMBER THAT!!!!!!!!!!!!!!! KNOBS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Okay.. i’m no IT person however I maintain our website hosted by you. I have read your emails, etc., but I can NOT upload or log into my site and it is need of updates. My boss is not happy with me. I don’t know my password cause you changed it and didn’t tell me. I use SmartFTP to maintain the site. I can not access my site through that software as I generally do and I believe it has to do with your security issues. This is day two and several attempts to get a message to you about this problem. You have no phone number for me to call you and I need to get this straightened out today. I can’t log into my account because I DON’T KNOW MY PASSWORD THAT YOU CHANGED. Please call or email me my new password ASAP. 908-789-0710

i can’t log in to see what i’m on.. and my site is NOT DOWN.. .IT IS fine.. but i can not access it or log in at dreamhost… like i said i’m no IT person and i’m really screwed here.. i have no tech person and a really mad boss,……….and no clue what to do to get to my site to update it.

I’m trying to launch a business here and I can’t update my damn site!

We were up for a while, but now we are back down again!

Everything on the ps12997 server is down for us. Anyone else on that web server having problems? I even upgraded the memory for our VPS by 50% and still nothing…

Well I’m finally back up. but still do not see any explanation as to why i couldn’t access my ftp when i was not on torr

I supposed RAID cards going down is a common excuse. That was my issue a couple days ago and I’m down again today. Wondering if it’s another RAID card… pttthhbbbttt… Dreamhost used to be a dream.

Hey everyone. We just closed our account and moved to Amazon. I suggest that all of you do the same. Cost about the same and its faster with much better performance and even support. Ciao, NightmareHost!

All my sites are on magneto and it won’t let me login via ftp to any of them, but the sites are all up (probably running off backups.) I’m not too concerned with this as long as I can get some confirmation that the problem will be taken care of very soon.

My site is still down!!! I expect a refund.

Is anyone else getting this ftp error?

“An FTP connection to (servername) could not be opened because the connection was refused.”

My sites are FINALLY back up but I’m still unable to login via FTP.

All of my sites and databases are down, in what must be the fifth time in a week. Amazon is looking more inviting…

For those that are having FTP problems it is not related to this incident. Instead Dreamhost had an security breach the other day and reset all FTP passwords. That means if you have not setup a new FTP password you won’t be able to use FTP until you do so. Go to https://panel.dreamhost.com/ (or try https://panel.dreamhost.com/index.cgi?tree=users.users&amp;) and select on the sidebar “Manage Users”. Then select the user you want to edit. In the configuration screen it will give you the option to set a new password.

Yet again our site is down. Is anyone else experiencing problems?

Our site is still down! Not very impressed with Dreamhost at the moment. Will definitely have to consider moving to a different host as this does not seem to be a reasonable option for operating a business.

Did the whole company just collapse?

I’m flying out to Brea. Going to knock down your f’ing door and I’m either coming home with my data or some form of justice.

22 of our client’s websites have been down for almost 36 hours. No updates on this blog regarding the status. Unbelievable.

You have put our small young company at serious risk.

So just made payment for renewal of my account… an now I can’t access it???? how bout a refund and we will call it a day as it seems as though the hosting part of “DreamHost” is just a dream…. at least you could have someone answer the people here..give some updates on progress or lack of…to much to ask … or is that broken too…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Degraded performance on personal backup server hanjin (RESOLVED)
4:41 pm

Degraded performance on personal backup server hanjin (RESOLVED)

Posted (January 22nd, 2012 at 4:41 pm PST) by danwy

The hanjin personal backup server has been experiencing some recent stability issues, which includes access difficulties and timeouts. Some solutions have been discussed by our admin team, but we haven’t decided on the best plan of attack yet. We wanted to put up this post to let our customers know that we are aware of the problems with this service, but unfortunately we don’t have an ETA on when they will be resolved at this time.

To clarify this post is regarding the personal backup server we offer as an additional service for all our customers. More information about what this service is for is on our DreamHost panel. This post is regarding only about the difficulties of accessing or staying connected to that personal backup service, and is not related to any other services or machines. This is not related to backups of your web, mysql or email services.

Update Jan 25th, 2:50pm PST: The two changes we made yesterday proved to be successful. We are now confident to close this case; all feedback from customers have been positive indicating the personal backup service is running again as expected. Thank you for your patience while we were working on this.
If you’re still having issues accessing or using the service while logged in, it’s most likely unrelated. Please contact support with the details, so we can look into it for you. -Andrea

Update Jan 24th, 2:00pm PST: We found two possible causes of the issues you’ve been having with the personal backup service. We eliminated both for now, and we’re putting measures in place that will prevent them from happening again.
If you’re are still having issues, going forward from this point, we would like to hear about it Please contact support with the following:
– name of your personal backup user
– the exact command you’re using
– the exact error(s) you’re seeing when attempting to use the above command
– how often does this happend, what times of the day
– does this affect smaller or just larger transmissions
– any logs you may have from your client
– traceroute results to hanjin including your originating IP address

Thank you for you patience while we resolve this issue. – Andrea

Update Jan 24th, 12:10pm PST: We are continuing to test the connection to the personal backup server and attempting to replicate the issues our customers are reporting. In order to better investigate this, we need the following details from those of you who are still having issues. Please contact support with the following:
– name of your personal backup user
– the exact command you’re using
– the exact error(s) you’re seeing when attempting to use the above command
– how often does this happend, what times of the day
– does this affect smaller or just larger transmissions
– any logs you may have from your client

Thank you for you patience while we resolve this issue. – Andrea

Update Jan 23rd, 4:45pm PST: We’re currently running connectivity tests with a fairly large file, trying to replicate and analyze the issues as they appear. We appreciate your patience and understanding while we’re working towards the resolution. – Andrea

Update Jan 23rd, 12:20pm PST: We are working with our Admin team on a definitive solution to this issue. We are handling issues with access to the personal backup server as they come in, but we expect to have a real solution to this issue very shortly! We will update this post as soon as we have details on this! Thank you for your patience in the meantime! -Patrick

This entry was posted on Sunday, January 22nd, 2012 at 4:41 pm and is filed under FTP Outages, General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

25 Responses to “Degraded performance on personal backup server hanjin (RESOLVED)”

I hope the backups are backed up ;)

What does this mean?? My websites are GONE… all the files are missing. I have no idea what happened – but I believe it is related to the outages over the last few days. So… I tried doing a restore to get my site back up since I CAN’T REACH ANYONE AT DREAMHOST and the restore FAILED.

My site has been gone for hours and I haven’t gotten any reply from Dreamhost despite numerous support tickets. Not one word.

We have not been able to upload anything for 72 hours. Before that was useless upload also.
Last night a simple folder listing took a whole night. Is a waste of our time. We know it’s not our side.

This is serious. Upload guarantee … does DreamHost have a service level agreement with their supplier?

We know what we are doing – SFTP connection and upload is not difficult.

To have issues for almost a week without any rectification …except for no ETA / theories is mickey mouse.

And by “stability issues” do you mean overwriting our sites with two-day old data, files, and configs? Cuz that is the instability that I’m seeing!

So that’s what happened… After several years of non-use, I finally decide to use my backup space for something, and the server goes tits-up. Awesome!

Since there seems to some confusion (in the comments above) about what this is talking about, the hanjin backup server is what you get access to when you activate your Backup ID on a DreamHost web hosting plan. No guarantees on data loss or anything like that, but the one place on DreamHost that allows personal and web site backups to be stored.

I’m just a customer, but I find it very useful. DreamHost doesn’t backup your accounts there.

I’m on hanjin and this really isn’t a big issue. As Jon Pearkins stated, hanjin is a “backups only” server. When setting up your user account for it you are also clearly told it is to only be used as a “backup of YOUR backups” and NOT a single backup source — there are no guarantees about the validity or stability of the data DH stores for you there.

My backups on Hanjin are the final link of redundancy in my home backup setup, so while it may be an inconvenience, barring my home exploding (and I’d be tits up anyhow…) my world will not end without it / while it has problems.

ALL MY SITES HAVE BEEN DOWN FOR 7 HOURS and I have not received 1 word from Dreamhost. I can’t reach anyone. Please explain to me how this is acceptable.

the likelihood of anyone receiving a personal response from dreamhost on a weekend when they are in the middle of a large scale recovery is very, very slim.

I would rather they work on fixing the multitude of problems that erupted after finding someone hacking into the accounts on their system and closing them out than consoling you for missing 7 hours.

I haven’t been able to access anything since Friday. Posting out of frustration won’t solve any problems, nor make you feel better.

well ssh/scp Login doesn’t work at all :(

@Gerry, Oh it does make me feel better! :-)

At last they acknowledged the problem. I was wasting my time last week trying to tell them there is a BIG issue with the backups server. I hope they fix it soon.

I was able to upload my second podcast episode but the feed is not working. Any ideas???

FTP login on ssh/scp wasn’t working for us but I got access by logging into dreamhost, went to manage users, clicked – open FTP server. It opened and I could upload. Hope this helps.

Did you file a support ticket?

Did you file a support ticket?

Etc.

Peace,
Gene

I filed multiple support tickets. wtf dreamhost.

My EGGS are in your hands. Please don’t fail.

not work tickets and not the CHAT
ps49734 and my machine is not listed, I’m no FTP / SSH what I do?

not work CHAT and not the tickets
ps49734 and my machine is not listed, I’m no FTP / SSH what I do?

How the hell can a hosting company be so inept at keeping things running? You all need to quit and to back to school to become a meterologist where 30-50% will be aceptable.

If I don’t hear from support within a week im filing a lawsuit to sue for my domain transfer.

I’ve never seen Dreamhost run so poorly.

For pete’s sake, would you folks that have web-hosting issues take them to DreamHost and leave this thread (about a backup server you probably don’t even use) to the folks that actually use the backup server? Spamming us doesn’t do you any good at all.

Hi Guys,

No update since yesterday. My backup last night started, but then failed. Any new updates?

-Jeff

Havent been able to log on in two days. Access and downloading problems have been going on for about 10 days. And yes…where are the new updates?

As of the latest announcement, performance when I login to the backup server via FTP seems back to normal. It listed a huge directory, containing several hundred files, in a couple of seconds. Download speed of a 75.6MB audio file seemed reasonably quick.

Peace,
Gene

Seems to be normal now, at last!!!

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘albali’ under file system check and being moved to new hardware (RESOLVED)
3:44 pm

Shared web server ‘albali’ under file system check and being moved to new hardware (RESOLVED)

Posted (January 22nd, 2012 at 3:44 pm PST) by andrea

Those of you on shared web server albali have been experiencing on-and-off downtime today. We’ve been trying to fix the machine, but decided that a file system check was necessary when the machine came back up as read-only (basically, mostly unusable for our customers). While that file system check is running, we’re also currently initiating a move to new hardware based on last backups, so we can serve sites in the meantime.

Please note, this is regarding the shared web server albali, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We apologize for the inconvenience this is causing. Please check back here often for updates. – Andrea

Update Jan 26th, 1:00pm PST: The restore is now complete and all sites should be up and running. Should you have any issues with your sites being down or missing data, please contact our support team and we will be glad to look into it for you.  – Oscar

Update Jan 26th, 5:15am PST: This restore is continuing to run smoothly. All sites should be up and running as well. Please contact support if you are experiencing any issues with your hosting on this server! – Chalise

Update Jan 25th, 10:16pm PST: This restore continues to run smoothly. All sites should be online. Please be sure to contact support if you are experiencing any issues with your hosting on this server! – Dan J

Update Jan 25th, 1:30pm PST: This restore is 79% complete and continues to run smoothly. Please be sure to contact support if you are experiencing any issues with your hosting on this server! – Seohee

Update Jan 25th, 10:30am PST: This restore is 76% complete and continues to run smoothly. Please be sure to contact support if you are experiencing any issues with your hosting on this server! – Patrick

Update Jan 25th, 7:49am PST: This restore is 74% complete and continues to run smoothly. Please be sure to contact support if you are experiencing any issues with your hosting on this server! – Chalise

Update Jan 25th, 5:30am PST: This restore is 72% complete and continues to run smoothly. Please be sure to contact support if you are experiencing any issues with your hosting on this server! – Chalise

Update Jan 24th, 10:25pm PST: This restore is now approaching 68% complete! Please contact support if you are experiencing any issues with your hosting on this server! -Dan J

Update Jan 24th, 5:15pm PST: This restore is now approaching 64% complete! Please contact support if you are experiencing any issues with your hosting on this server! -Seohee

Update Jan 24th, 2:03pm PST: This restore is now approaching 62% complete! Please contact support if you are experiencing any issues with your hosting on this server! -Patrick

Update Jan 24th, 10:42am PST: This restore is ongoing and is approaching 60% complete! Our Admin team may be moving this server back to the original hardware if the checks we are running are passed successfully, so we will update this post if that happens. Please contact support if you are experiencing any issues with your hosting on this server! -Patrick

Update Jan 24th, 7:15am PST: The restore is now 54% complete and continues to run smoothly. As previously reported, our admin team is still looking into possibly moving it back to the original machine if the hardware can be fixed. All services should be up and running, so please contact support if you are experiencing any issues! Further updates to come. – Chalise

Update Jan 24th, 3:45am PST: The restore on this server is continuing to run. As previously reported, our admin team is still looking into possibly moving it back to the original machine if the hardware can be fixed. All services should definitely be online, so please contact support if you are experiencing any issues! Further updates to come. – Chalise

Update Jan 23rd, 10:36pm PST: This server is still being restored. The admin team is still looking into possibly moving it back to the original machine if the hardware can be fixed. All services should be online, so please contact support if you are experiencing any issues! We’ll continue to provide you with updates once we receive further information. -Dan J

Update Jan 23rd, 5:46pm PST: This server is still being restored and is approaching 42% complete! The admin team is currently looking into possibly moving it back to the original machine if the hardware can be fixed. All services should be online, so please contact support if you are experiencing any issues! We’ll continue to provide you with updates once we receive further information. -Seohee

Update Jan 23rd, 1:10pm PST: This server is still being restored and is approaching 40% complete! All services should be online, so please contact support if you are experiencing any issues! More updates later! -Patrick

Update Jan 23rd, 10:00am PST: Your users and sites on albali should still be up and running, and the restore continues to run. We will update with further details in the next few hours. If you experience any issues with your sites, please contact support! -Brandon

Update Jan 22nd, 4:15am PST: Your users and sites on albali should still be up and running, and the restore continues to run. We will update with

further details in the next few hours. If you experience any issues with your sites, please contact support! – Chalise

Update Jan 22nd, 8:52pm PST: Your sites are still being serviced by the secondary machine, but otherwise everything is running as expected. The restore is currently at 13%. We’ll update this post with any status change we receive from our techs, but if you have any issues with your site on albali, please contact support with the details. – Seohee

Update Jan 22nd, 6:30pm PST: Your sites are still being serviced by the secondary machine, but otherwise everything is running as expected. We’ll update this post with any status change we receive from our techs, but if you have any issues with your site on albali, please contact support with the details. – Andrea

Update Jan 22nd, 4:38pm PST: The secondary machine is now up and running and serving sites from the last backup. We’re still checking the original web server to make sure there is no file system corruption. We’ll update this post with more information soon. – Andrea

This entry was posted on Sunday, January 22nd, 2012 at 3:44 pm and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Responses to “Shared web server ‘albali’ under file system check and being moved to new hardware (RESOLVED)”

hello.. i can’t log in cause i don’t know my new password and i can’t contact support because i can’t log in……..
send email with new password plz

I have the same problem as Christina… I would really like a response from support. I have no clue on all the types of servers etc, so don’t know if this post is affecting me or not. But PLEASE respond! I’ve left a comment on another post earlier today, tried to send an email (it bounced back) and asked on Twitter what to do to get into my account to change my passwords. No responses. You all just took money out of my bank account on Thursday. I think you should respond. I’ve never had a problem with you all but I’m starting to doubt your customer service at this point.

All my websites are down, not just http://52weeksofkink.com. NO notification from Dreamhost and no response to my email. I am pretty frustrated.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver233 moved to new hardware and restoring from backups (RESOLVED)
1:07 am

Web VPS host homie-vserver233 moved to new hardware and restoring from backups (RESOLVED)

Posted (January 22nd, 2012 at 1:07 am PST) by jj

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver233′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver233’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 29th, 5:15pm PST: We have confirmed with our admin team that the restore for this server has completed. If your sites are missing data that means the restore did not find anything in the backups automatically.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help see if there is anything else we can do for you. -dan w

Update Jan 28th, 11:15pm PST: All VPS guest machines are online and responding. The restore process is running and cpu load on the host machine remains low and stable. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – JJ G

Update Jan 28th, 3:30pm PST: All VPS guest machines are online and responding. The restore process is running and cpu load on the host machine remains low. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – Oscar

Update Jan 28th, 5:30am PST: All VPS guest machines are online and responding. The restore process is running and cpu load on the host machine remains low. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – JJ G

Update Jan 28th, 1:30am PST: All VPS guest machines appear to be online, mounted via backups, and responding properly. The restore process is running and cpu load on the host machine remains stable. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – JJ G

Update Jan 27th, 8:48pm PST: All VPS guest machines appear to be online and responding properly. The restore process is running and cpu load on the host machine seems low. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – Dan J

Update Jan 27th, 3:25pm PST: All VPS guest machines are online and mounted via backup machines. The restore process is running and cpu load on the host machine seems low. If your sites are missing data that means the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – Dan J

Update Jan 27th, 12:50pm PST: All VPS guest machines are now up and mounted via backup machines. The restore process is still ongoing, so if your sites are missing data the restore has not gotten to your server just yet.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help! – Oscar

Update Jan 27th, 4:00am PST: The restore process continues to run without any issues. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! – Chalise

Update Jan 26th, 10:00pm PST: The restore process continues. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Dan J

Update Jan 26th, 6:19pm PST: The restore process continues to run. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Dan J

Update Jan 26th, 4:56pm PST: Our Admin team has finished working on the few guests that would not run properly and have now restarted the restore process on the host! If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Patrick

Update Jan 26th, 1:30pm PST: The restore is running without any issues at this time. There are a few VPS guest machines that our admin team is working on which are having some issues. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! – Oscar

Update Jan 26th, 5:05am PST: We managed to significantly lessen the load on the server overnight which should help move restores along and any alleviate any slowness you might have been experiencing. Our admins are still working on restoring a few servers with missing data. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! – Chalise

Update Jan 25th, 10:19pm PST: The restore process seems to have increased the CPU load on the server and that is causing some slowness. Our admins are working to correct that. We are currently restoring a few servers. So if you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Dan J

Update Jan 25th, 1:26pm PST: The restore is ongoing and with the exception of one VPS guest, services are online! Our Admins are working on fixing up the last guest that is not online yet, and will then restore all guests to make sure we’re up-to-date on backups. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Seohee

Update Jan 25th, 10:34am PST: The restore is ongoing and with the exception of a handful of VPS guests, services are online! Our Admins are working on fixing up the guests that are not online yet. If you find you are missing data, or have old data on your VPS, please contact support and we will be happy to help! -Patrick

Update Jan 25th, 7:43am PST: The restore continues and all servers appear to be online. We are noticing a few guests that were not properly restored and our admin team is looking into this. If you do seem to be missing data or have issues with any sites on a VPS on this host, however, please make sure to contact support! – Chalise.

Update Jan 25th, 5:15am PST: The restore continues and all servers should be online. If you seem to be missing data or have issues with any sites on a VPS on this host, please make sure to contact support! – Chalise.

Update Jan 24th, 9:51pm PST: The restore is back in progress. All server should now be online.  We will provide more updates as soon as possible. Thanks for your patience! -Dan J

Update Jan 24th, 3:04pm PST: The restore is back in progress but there are some guests down at this time. Our admins are working on getting them back up and running now. We will provide more updates as soon as possible. Thanks for your patience! -Seohee

Update Jan 24th, 12:57pm PST: We are struggling with some issues with the new hardware for this server and this restore is currently stalled. Our Admin team is working on getting this restore moving and we will provide more updates as soon as possible! Thanks for your patience! -Patrick

Update Jan 24th, 9:43am PST: Our admins are still prepping the new hardware. They will begin the migration withinin the next 30 minutes or so. Please note that there will be a brief period of downtime during the migration. However, it will be well worth in the long run. Since your service will be running on shiny, new hardware.

We will continue to provide updates as information is made available. Please check back. – Brian S

Update Jan 24th, 5:18am PST: Our admins will be moving this machine to new hardware shortly. This may cause brief downtime for guests once it occurs. The move has not yet begun, and we will update with further information once we have it. – Chalise.

Update Jan 23rd, 10:00pm PST: Our admins were able to repair the original hardware, so the guests have been moved back to it, and the original hardware is running well.  The admin team are still looking into whether or not a move will be required to new hardware, and don’t believe it will be as long as the original hardware remains stable. After some monitoring the decision will be made. If it is to remain on the now fixed, original hardware, any new data uploaded to your guests during the move and restore will be copied over to your machine. We will make sure to keep you updated as soon as we have further information. -Dan J

Update Jan 23rd, 6:00pm PST: We have new developments regarding the move to new hardware. Our admins were able to repair the original hardware, so the guests have been moved back to it, and the original hardware is running well.  They are still looking into whether or not a move will be required to new hardware, and don’t believe it will be as long as the original hardware remains stable. After some monitoring the decision will be made. If it is to remain on the now fixed, original hardware, any new data uploaded to your guests during the move and restore will be copied over to your machine. We will make sure to keep you updated as soon as we have further information. -Seohee

Update Jan 23rd, 5:12pm PST: Details regarding the status of this VPS host are being looked into at this time. We will update this post as soon as we have further information! -Seohee

Update Jan 23rd, 1:31pm PST: This restore continues! We will have more specific details on this one within a couple of hours! Thanks for your patience! -Patrick

Update Jan 23rd, 7:20am PST: We came across a few issues getting the VPS guests on the host machine online last night. The restore process is still temporarily halted. Fortunately, we were able to move the host machine back to its old hardware to bring everything back up until we are able to continue the restore process. All VPS guests should be running again at this point. We will continue to update with further news as our admin team continues their work on this host machine. – Chalise

Update Jan 23rd, 2:30am PST: Our admins are still working on getting the VPS guests to boot. Unfortunately, we don’t have an ETA on when that will be at this time, though we are actively working to get things back up as soon as possible. This is also causing the restore to be temporarily halted. Further updates to come. – Chalise.

Update Jan 22nd, 8:43pm PST: Our admins are having issues getting all of the VPS guests to boot due to the network file system running into problems. They are still working on it and hope to have all the VPS servers on this machine up as soon as possible. Unfortunately, we don’t have an ETA on when that will be at this time. Due to the problem with the network file server, the restore has come to a temporary halt. We’ll update this post again shortly with more info. -Seohee

Update Jan 22nd, 5:45pm PST: Our admins are still having issues getting all of the VPS guests to run correctly on this machine. They are still working on it and hope to have all the VPS servers on this machine up shortly. Unfortunately, we don’t have an ETA on when that will be at this time. In the meantime, restores continue to run to it in the background. We’ll update this post again shortly with more info regarding those problematic VPS’s. -dan w

Update Jan 22nd, 3:30pm PST: Looks like we spoke a little early. Shortly after the last status post update, the restore process failed. Upon investigating that, we found a configuration error that was preventing Apache from starting on some of the VPS servers on this host. We’re running a loop command now to configure all the Apaches on this server and get sites that are hosted under them up and running again. We’ll update this post again shortly once all the configs are complete and we’ve confirmed all Apache’s are running. -dan w

Update Jan 22nd, 2:30pm PST: This restore process continues to progress with no problems. It’s still early in the restore process, so we can’t confirm actual percentages, but data is being copied over at a fast rate. Sites should continue to respond during this process and we will continue to post further updates until the process is complete. -dan w

Update Jan 22nd, 10:00am PST: The file system check on the original hardware has completed, but our admin team has made the decision to keep the server on the new hardware in the interest of long-term stability. As stated in the main post above, the data on the backup server will be mounted directly over NFS, so no sites should be down while we’re restoring the files locally to the new hardware. If you notice problems with any site on this server, please contact support to have it investigated. -dan w

Update Jan 22nd, 8:00am PST: The file system check is now on pass 1B and continues to execute. We will continue to post further updates. — JJ G

Update Jan 22nd, 5:23am PST: The file system check is still on the first pass but is now 58% complete. We will continue to post further updates. — JJ G

Update Jan 22nd, 4:28am PST: The file system check on the original hardware had to be restarted. The process has been restarted and is currently on the first pass with 2.14% done. We will continue to post further updates. — JJ G

Update Jan 22nd, 02:15am PST: VPS guest machines are just starting to come up on the new hardware, while the file system check continues to execute. We will continue to update this post as these processes continue to run. — JJ G

This entry was posted on Sunday, January 22nd, 2012 at 1:07 am and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

33 Responses to “Web VPS host homie-vserver233 moved to new hardware and restoring from backups (RESOLVED)”

The following machines are affected:
ps13998 ps14999 ps15796 ps18261 ps18306 ps37782 ps47511 ps47512 ps58146 ps58862 ps58867 ps58869 ps58909 ps59017 ps59081 ps59390 ps59468 ps59485 ps59540 ps59774 ps59817 ps60048 ps60196 ps60321 ps60503 ps60631 ps60816 ps60822 ps62707 ps63103 ps63833 ps64687 ps64715 ps64856 ps64950 ps64959 ps65207 ps65306 ps65362 ps69703 ps69982 ps72217 ps73929 ps76665 ps76882 ps78052 ps78374

January 22nd, 2012 at 8:31 amMonticello Live Says:

@jj can you fix my site http://www.monticellolive.org pleases

my website is down, can’t get my emails either.
What’s up with you guys???

How much longer? This down time is killing us!!!!!!!!

So I guess today was not the day to sign up and start creating my website? It worked for two seconds and now its dead. Did I do something wrong or is this related?

After resolving my email issue – at least I thought I did – last night, now it has stopped working. Webmail is not working either. What is going on with email? Can you just tell me in plain English –that would be helpful.

After finding someone hacking my wordpress blog consistently over the past few weeks, I can understand the concerns and magnitude of this. Thank you Dreamhost guys for acting on what could have been a much worse situation (i.e.: everything going down without anyone being aware of it, and discovering it without backups)

Anyone who complains about the methods hasn’t been hacked, or found redirects form something as simple as a file included with a wordpress theme that affects your database or important files. I’ve since deleted wordpress due to the packability of their open nature.

And yes, I am affected, worse than most. I can’t get back online until someone who received the ‘reset passwords’ email resets mine and lets me know what the new one is. I’m willing to accept the lack of control until everything is back online. Patience, people.

‘hackability’. stupid spell checker.

My site is still down despite DreamHost stating above “so no sites should be down while we’re restoring the files locally to the new hardware”. Really????????? I have been down since approx. 3 a.m. EST this morning and it is now 2:10 p.m. EST – almost 12 hours. I have no more patience!

I just signed up a new account and cannot continue. Did I sign at the wrong time?

@Julia. I feel your pain. I’m on another server. My site has been down going on SIX DAYS! Unreal.

My server (ps56563) is not listed and i can’t access it… “Your home directory does not exist on your web server” for all users in that machine :| What should i do?

My panel does not work tickets and not the CHAT

ps49734 and my machine is not listed, I’m no FTP / SSH what I do?

not work tickets and not the CHAT

ps49734 and my machine is not listed, I’m no FTP / SSH what I do?

Been down for four days now. Like silverfox, our vps (7792) isn’t listed.

No online chat, no ticket answered, more than 12 hours offline. Unacceptable.

I had to set dns to change main page to a warning page out of DH servers because I couldnt do it in my DH VPS server. Unacceptable.

No support, no eta, nothing. Only a bill next ending month. Unacceptable.

“Please note that there will be a brief period of downtime during the migration”

Is 16 hours a brief period of downtime? Im loosing a lot of money with that.

I got a message saying I was back online and everything looked OK. Unfortunately, we’re not and all of my folders on the FTP only have the favicons in there. Anybody else have that issue?

Can we get an update on this please?? My sites have all been down for the whole day (and much of Monday and Sunday as well) We all deserve another update!!!!

“We are struggling… Thanks for your patience! -Patrick”

Patience? Whats that? I dont have it anymore so, dont thank me for it.

“All server should now be online”

Wrong answer! My site is down for more than 24 hours! I cant believe DH is doing that!

Our website continues to be down. Please provide an ETA of when all files will be restored.

22 of our soon to be ex-client’s websites have been down in excess of 36 hours. Dreamhost has ruined us.

2 whole days down.

Congratulations DH. You got another ex-client. You are a nightmarehost. I loose a lot of money with you.

DH never more!

My sites got back up last night, but none of our forms work. Anybody else have that issue?

They moved my domains to another server. Now, they are very, very, very slow. Nobody can use it. Great solution!

I cant believe Im still with problems with nightmarehost. Im going to third whole day without my domains up.

Unacceptable.

I must be that ONE guest that’s still down – because my sites have NOT been restored. It’s been 5 DAYS and my site is still down. Not only is it down – but it is physically MISSING from the ftp account which makes it impossible for me to move the site elsewhere.

Also, Dreamhost support has done nothing but tell me inaccurate information (“your sites have been restored” – no they have NOT) and then continue to tell me “it will be another few hours your VPS machine is still being restored” which I’ve been told more times than you would even believe.

5 days is NOT a few hours.

I would expect at the very least a personal phone call from a supervisor for having my site down for 5 DAYS but apparently Dreamhost doesn’t feel that way.

And now… ALL OF MY SITES ARE DOWN.

Yes, everything bad is back!!!

Congrats again DH! All my domains ARE DOWN AGAIN!

Maybe all the folks that are “losing money” should host their sites elsewhere.. DH is a “budget host”. You get what you pay for.

Oh my god!!!
Whats going on? Please help me. I have got only 3 site on one VPS and this site is very important for me. Server is still down.
Could you please tell me when it will be run again?

FTP ERORR
Status: Connection attempt failed with “ECONNREFUSED – Connection refused by server”.
Error: Could not connect to server

I was tryed the account test: Your website does not appear in the Apache configuration file.

jaguars server
FTP ERORR
Can not connect to FTP.
Pleaes Help me

AGGGHHH, all my servers are down, I’m losing money and SEO, I’ll have to go back to work at Target!

Just kidding.

Let’s start a group erotica slashfic. “Billie, a young man/woman with a 12 inch member and 38DDD breasts, walked into the biker bar swinging a whip…”

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Password changes delayed in the web panel (RESOLVED)
9:22 am

Password changes delayed in the web panel (RESOLVED)

Posted (January 21st, 2012 at 9:22 am PST) by Oscar

Processing user updates is taking long than usual due to the of sheer number of customers requesting password changes on our system. We understand your desire to get things working in an expeditious manner and we are working hard to get you there. We’re examining ways of decreasing the queue depth but we’re still faced with the fact that there is a considerable amount of work to be processed and apologize for the delay.

Update Jan 22nd, 9:12pm PST: We’ve been keeping a close eye on our panel update times and it looks like things are back to normal at this time. If you experience any delays, please contact support. -Seohee

Update Jan 22nd, 2:30pm PST: We’ve been keeping a close eye on our panel update times and it looks like things are getting back to normal at this point. Update requests are getting back to the usual 15 minute wait time to be processed. We’ll keep this post active for a short while longer while we continue to monitor things, but expect to be able to comfortably resolve this issue soon. -dan w

Update Jan 22nd, 11:30am PST: Password changes are slowly getting caught up. According to our stats it looks like they’re all being processed in under an hour now. It’s a little higher than normal still, but is better than it was. Also, at some point after the initial notification went out asking customers to reset their passwords went out, the passwords were reset again. Unfortunately, we had no way of telling which passwords were already changed, so to be safe we forced out another change. If you changed your password and it stopped working again, this is likely the reason. At this point, any password changes made should be pushed out within about an hour. If you’re still having problems accessing your FTP now, please submit one last password change and give it an hour or two to update. If it isn’t working after that, please contact support to have it investigated further. -dan w

Update Jan 22nd, 07:30am PST: Due to the number of password changes currently being pushed through there is still a delay in getting those updated. — JJ G

Update Jan 21st, 08:30pm PST: Password changes are looking better now, however there may still be some delay in getting those updated. Hang tight, though! — Brian H

Update Jan 21st, 11:00am PST: Password changes are still processing. albeit slowly. At this time we estimate that any password change will take approximately 1-2 hours to fully update in our system. We sincerely apologize for the delay in these changes taking effect. -Oscar

 

This entry was posted on Saturday, January 21st, 2012 at 9:22 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

89 Responses to “Password changes delayed in the web panel (RESOLVED)”

can somebody answer the support tickets or enter chat support? All files from my websites have been deleted and I need somebody to explain to me what is going on.

I changed my passwords and cannot access ftp! WHEN WILL IT WORK?!?! Clients are mad, I’m behind deadline thanks to this and now nothing works.

I REALLY NEED SOMEONE TO ANSWER, I cannot access FTP and it seems people are having SERIOUS issues with their files. Y’all BETTER GET ON THAT CHAT!!! It seems lately dreamhost just gave up on live chat and customer service. we all better get coupons!!! I have a balance right now I would really like to negotiate before I pay, since I technically just paid for hosting 4 times this morning with this mess and losing money to clients. YAY!

Nikos, if you are referring to leap.gr then it is functioning at the moment …. just sayin’

Thanks for posting this. I was wondering what was going on.

So, how long do you anticipate it taking? Hours? Days?

Is this for FTP password changes only? or all password changes?

Thank you

Imagine that… forcing a password resets on thousands of users at the same time would cause thousands of people to change their passwords at the same time.

Someone didn’t think this through.

What is the issue here, do you not one way encrypt your passwords? How do you store them? This is very unsettling.

Could you please publish, either here or on the manage page, an estimate of how long it is currently taking for a password change to take effect. The way it is now, it makes me wonder if this whole thing is a giant phishing scam. And there’s no way I’m changing my email passwords if this is how it’s going to go. Think this through and phase in change requests.

No wonder Dreamhost is starting to look like a Circus, Simon Anderson, your CEO, is a lawyer.

Can’t speak for anyone else, but GoDaddy would have told me that this was all my fault. Give them time – stuff happens. They’ll get everything working again.

Just a quick question for Dreamhost. I noticed that my VPS passwords from before the hack still work. Does that mean that VPS users weren’t affected, or just that those passwords haven’t been reset automatically?

@Jahojo I moved the site to another provider, it was too crucial to stay down, and wait for DreamHost to even answer me what the fuck is going on.
Thanx :)

it’s ironic that i just move all my hosting to dreamhost and then BAM!

breaches happen, the point is that you guys reply more often with status updates…
i have 6 site down with no concept of time-frame or the severity of the issue.

please let all of us know, yea?

you guys really funked this one up *slow clap*

Can i sue them for this? for lossing money?

they even wont tell when can i change my password, im losing some REALLL MONEY in here!
i swear, if all this wont be fixed in the next hour, im leaving!

All my sites are down for over 10 hours now, who’s gonna pay for this incredible loss? They can’t even keep us posted on what’s going on?

Come on guys! this attitude you don’t help to resolve the issues.

I know you’re concerned about it, and me too! but posting comments in order to try to press the staff don’t resolve nothing.

Don’t forget, this are machines and they are human beings trying to do their bet to resolve it asap.

Just think about, and thanks for read.

@sebastian
you are right, but their doing that for much more than that it needed!

and why they reseted my password if i didnt asked to?
why they just couldnt send me an email that alert me to change passwords?

now all my clients are shouting at me… and im the one who losing money!!!
and im pretty sure that dreahost wont give me that money back!

.

Step 1.) Tell people they should use you as a hosting provider, even though you can’t keep FTP passwords secure from hackers.

Step 2.) Take people’s money.

Step 3.) Not refund anyone for their frustration when you force users to change passwords because you let a hacker get access.

Step 4.) Not give a fuck and still keep money, thinking that just telling people what is going on is enough to satisfy them.

.

Yup. Migration time.

@avihay @Herp Derp

Both are right!, and totally agree with you guys, i’m lossing money for every hour, i hope, when all this are completly resolved, discuss the “gift” for this big trouble, but for now, i only can whait and send “good vibrations” if its help to anyway

Have good day!! ( if its possible, for sure )

Upset. Told to change password, but not that password changing would take (hours, days, weeks??) extra time. Why tell us all to change our passwords ASAP if there isn’t the manpower to do it?? Ridiculous.

wow, just had a cartoon on in the background, character is in school and says, “hmmm, my locker combo’s changed, good thing i have this paper clip.(picks lock). oh no! where’s all my stuff! this is strange, what’s going on today?”

haha

I don’t think a few hours is too bad considering the implications of a security breach (which happens all the time to all kinds of sites). I’d rather wait a few hours and know that my data is safe.

a few hours isn’t bad when we’re at least updated for those hours. there has not been one update to this post

i take that back, they just had an update

QUESTION: can someone tell me if i change my FTP password, is this going to change my databases from working that are tied to that “user”? i know like when youre setting up a cart or blog you need to associate the ‘user’ to the database and password—this shouldn’t change anything right? just my ability to connect to FTP?

January 21st, 2012 at 11:47 amWilliam Xavier Bobberson Says:

FTP? Once I paid a “lady of the evening” to give me an FTP behind a Donairs in Halifax. If was grrrrrrreasy.

I think I can say most customers will be expecting to see a report on what happened, how it happened, and what steps will be taken to ensure it does not happen again. Until that occurs trust and faith in this provider will be shaken.

How about not storing your passwords in a recoverable form in the database so that you don’t HAVE to process hundreds of thousands of password changes in the future?

I have worked with Dreamhost now for almost 10 years and in all of that time their customer support was the BEST of almost any other company (online or off) out there. This was not something that was expected and I’m sure what they are doing is to protect YOU!! So give them a break, let them do their jobs, and the system will be better for it. I’m sure they will find ways to strengthen it even more!

So you have choice to make. Be all pissy about it, or recognize that sometimes LIFE throws you a curve and you swing anyway. You’ll get your “home run” soon enough. But bitching about it isn’t helping ANY ONE!! Maybe this is the universe’s way of telling you to take a little time off and “smell the roses” so to speak.

Hi: I just got the warning email and followed the instructions to change my password for my FTP. Now I can’t connect to my website. It says Login error. Can someone help? I am not very good ast this stuff. Thanks

All these moaners, when I was coming here to say THANK YOU to the team for catching this. Hacked ftp passwords and all our websites replaced with spam would have been far worse!

So once again a BIG THANKS to the dh team for being on the ball here.

Rootin’ for you guys. What a nightmare this must be for you.

It’s back up!!! So all the bitching should stop! DreamHost is truly one of the best hosting companies I’ve EVER dealt with and their customer support is superior. Thanks guys for getting us back and running.

I’m glad you took swift action.

Who in gods name would put serious mission critical websites on a shred host anyway. Why all the bitching. Shit happens.

I retract my heated statements, i was overwhelmed with client calls and should not have posted. I have been with DH for sometime and over all i must say that they do a great job. things do happen and in the end, they are clearly working in our favor. sorry for the bad vibes!

Password changing is still quite slow but it seems to be well less than an hour at present.

My concern in all this is that if FTP/sFTP/Shell passwords were considered suspect then everyone who has a Drupal/Wordpress/Joomla or other CMS website with a config file needs to change their MySQL passwords as well. Anyone that can access your site in a sFTP/Shell and hacked the DH database would know that these files are perfect targets. Now on top of that you add the possibility of user tables in these CMS systems being accessed. Doesn’t matter if they’re one way encrypted since most users fail to follow simple rules so at least one of them probably used “password” or some variant. With a Rainbow table attack most hackers would make quick work of these.

DreamHost is claiming only FTP/sFTP/Shell passwords are suspect and no customer data was accessed. If that were true then why the over-reaction. I believe they, DH, have no clue what attack vector was used to compromise their system. Add to this the fact that VPS passwords have not been reset and the DH solution is not complete. The nightmare is far worse then claimed. All my passwords are auto-genrerated and at least 12-25 characters long and we only allow sFTP/SSH. I use enhanced security and I don’t use the one-click installs nor do I accept the defaults for anything in my CMS’s ala jos_ or wp_ prefixes for table names like the one-click installs do. All that extra effort is wasted. If DH does know the attack vector used then nuke that site and work with that customer to setup the site correctly.

This is not “bitching”, “moaning” or any other adjective. It is a legitimate concern for the security of my personal, my companies and our customers websites and users.

I have no plans to threaten moving from DreamHost to another provider, I personally steer folks to this company because I love the fact they support non-profits the way they do. But if some of you expect me to sit back, shut up and accept the story in this case, sorry it’s not going to happen.

I posted about the delay, but I was asking for help – not complaining. I thought maybe I was doing something wrong or maybe there was something that they needed to know about. Apparently it was just the sheer volume that slowed things down. I got an almost immediate answer which is pretty amazing considering the problem – I’m very impressed! And, of course, I just had to wait a while – nothing serious. Reminds of when I lived in Venezuela – you had to wait in lines for hours just to get anything done – ha! You’d think I’d have learned some patience ;-)

Fuck all this. I’ve been here for 6 years and I have the exact opposite experience of K.C.. This has been a quickly declining situation for a long time. Luckily I quit hosting here a long time ago for my customers and only use DH for test servers on occasion. These guys can’t even keep a fucking mail server up longer than a day.

Still no official reply, why all the files from FTP where deleted. Great support really after 6 years of using your services.

January 21st, 2012 at 2:08 pmPatrick Farling Says:

@fart: Go ahead and leave, then. No one is keeping you here. However, why you feel it is necessary to throw the proverbial trashcan around is beyond me.

@Nikos: Let me know your domain or submit a support ticket ATTN:Oscar and I will be glad to look into it for you.

Thanks to Dreamhost for all your hard work. I believe you are doing your best to fix this and will learn from it. We’ve never had a problem with you before this.

I must Chime in that DH support has treated me well. I would like to see more info coming as to what we actually need to change. Like John Wicks Says: Any of us with CMS’s might have to change DB passwords and that isn’t easy on the newer versions of good CMS’s who Hash the info. So what exactly beyond FTP and possibly email passwords do we need to update?

I am glad DH acted quickly and no site or server is 100% hack proof…but more info to account holders is warranted.

I changed all my FTP\Shell passwords about 7 hours ago and I still can not access my ftp.

I changed my MySQL password ~8 hours ago yet my wp-config is still using the old password.

My new SFTP password is working though.

Lost the business of a long time client that will cost me the same as 10 YEARS of hosting… EVERY MONTH … trying not to throw up

Thanks Dreamhost. I have gone through the agony of a site hack before, and I’m glad you took immediate action on this so I avoided having to go through that again.

@Christopher Now having read what the actual exploit was (see http://blog.dreamhost.com/), a legacy database remnant from years ago, verifies, for me at least, that this whole debacle is an over reaction. The DH solution was/is worse than the perceived threat. Also given the fact their solution in this case is incomplete, they reset Shared and PS accounts (created less than a year ago) but not VPS accounts (created more than a year ago), it, the wholesale password reset, doesn’t appear warranted to me.

So passwords were stored raw unhashed?

It’s been 24+ hours and still can’t access any FTP – across 3 VPSes!

My previous email to support took 10 HOURS to get a response. And I’m still locked out of our account. If this isn’t resolved by Monday morning, guess it’s time to move on to a company with enough customer support reps. Ugh.

I still cannon access FTP!!!! Why do you bother sending me these BS emails about how it’s going to be fixed in 2 hours!!! It’s been well over 24 hours since I reset my password. You could at least be honest about this!!!

It’s been over 4 hours for me. Please update… something… to at least let us know that you’re aware of this lengthy delay. Seriously. It’s not too much to ask.

@Dane, @Bfuddy, @Maggie – Send in a ticket Attn:Oscar and I can take a look at it for you and make sure the updates are pushed immediately.

I have no idea if the password reset went through for me but my ftp seems to be down in either case.

‘Connection attempt failed with “EAI_NODATA – No address associated with nodename”.’

Anyone else having this issue?

I changed my password this afternoon…. it’s now 12:37am EST and I still don’t have access. The thing that kills me is that I have a VPS and I was able to FTP into my server this morning. I saw that someone else with a VPS was also still able to FTP in to their server. They asked if they still needed to change their password – see the post from Atnevon dated January 21st, 2012 at 10:18 am. No one responded to him so I reset my password and I’ve been unable to accomplish anything all day today because of this. For all I know, because you didn’t reset my VPS password, my server was safe and I could have been working all day. I hope that I wake up tomorrow to find that my password change has taken place. Meantime, I’m sending in a support ticket to Oscar and going to bed hoping for the best.

I agree. I did the same thing, hoping to have FTP access back for projects for clients. I sent a ticket asking whether I should resubmit or wait. I am hoping the process will complete overnight and if not, will check on tickets tomorrow.

Poor Dreamhost guys having to spend their weekend fixing this mess. If you’re hot, tired, sweaty and of neither distinct gender, call me when you’re done.

Dear Sir,
We’ve been hacking (hack pharma) for about 4 months now and asked the hot line about this issue.
They answered us it was an update problemn of our softwares (wordpress, etc).
Every update has been done, malicious code has been cleaned up, FTP passwords have already been changed several times but everytime hackers come back again and again, putting their malicious code in our files.
I think there is a serious security breach on your server to deal with. It is not enough to only change passwords.
In this list of comments, I read above me (HarryB,Hector, Craig, Graeme, Chris, rich…), that other customers have experimented the same problem
Hope you can solve this problemn very quickly as it can’t be manageable on the long run,
Thank you very much,
Best regards

This is strange, I waited roughly 3 hours for passwords to update yesterday, which they did and worked perfectly for the rest of the day. Though today the new passwords wont work?
Has this happened to anyone else? It seems they have been reset again..

@Alex… me, too! OMG! This is sooooo upsetting. I have so much work to do. I planned to work this entire weekend and I’m in big trouble if I don’t get my work done for my clients.

what do we do? put new passwords? or has Dreamhost disabled FTP access? would be nice to have a new update for today from DH..

Alex: yes, I’ve had both FTP/Shell & Email passwords that were reset mysteriously go back to old values. Has your experience just been FTP passwords, or email too?

I’ve also experienced the mysterious password-reset problem. Changed passwords worked yesterday but not this morning. I’ve changed them again and that seems to have worked (I haven’t checked them all, though, so not all accounts may been properly updated).

Same here – can’t connect to FTP with new OR old passwords… hope to hear an update about the status soon, as i’m sure we all need to plan around this. just let us know what the deal is, please!

Yes I reset all my password yesterday and confirmed that the reset passwords were working.
Now today they don’t work.

I’m not upset at dreamhost, these things happen and i’m glad they reset the passwords, though it would be nice to see an update posted somewhere that says “the que of reset passwords is X accounts long, we anticipate that it will take Y hours/days before some users will be able to connect.”

i reset my pw yesterday along with everyone else in the hemisphere and it seems it hasn’t been processed yet. I just want to know when i should start worrying and when i should just open a beer and nestle in with netflix.

I have my own account here on Dreamhost and I host numerous websites for my clients on my own VPS. But in addition to that, I have several clients who have their own accounts on Dreamhost – and I manage their accounts. So… yesterday morning I woke up to learn – same as everyone else who uses Dreamhost as their host – that FTP passwords had to be reset. My first priority was my biggest client. He has a VPS as well. I reset his password and waited 10 minutes. No go. I waited two hours. Nothing doing. In the meantime, I tried to FTP into my VPS and – huh? – I was able to access my server and the numerous websites that I have on that VPS. But the email from Dreamhost said that they reset all of our password so it was rather curious that I was still able to FTP into my VPS. I was reluctant to reset my password because I knew that my other client’s password was still not working. I was really unsure of what I should do. Should I leave my FTP password as it was because it was working and I would still be able to get my work done? Or was I asking for trouble and leaving my websites vulnerable. Well… I have several other clients who pay for their own hosting on Dreamhost with their own accounts. So I went down the list and systematically changed all of their passwords. After I took care of all of that, I decided that it was best to reset my own password as well. Better to be safe than sorry. I attempted to change my password and I selected the checkbox to let Dreamhost come up with a random password for me. I clicked on “Save Changes” and the page just reloaded to the same page… there was no message above telling me what my new FTP password would be. I tried it again. Same deal. I tried it again and received an error message that my new passwords didn’t match? Well, how could that be when Dreamhost was selecting the password. So I tried putting in my own password. Then and only then did Dreamhost finally provide me with a new password – but despite the fact that I added my own password, Dreamhost gave me a random password that they picked for me. I was just happy that it worked but also a little worried about whether or not it would take.

By this time, my biggest client’s FTP password was now working. So I figured that the rest of them would eventually start to work. I checked my FTP connections throughout the day. Two of my other clients now had FTP access. Mine still wasn’t working. I worked all day and night on other projects and kept testing my FTP connection. By 12:30am EST, FTP connection to my VPS was still not working. I wrote a support ticket to Oscar (see above Oscar’s post at January 21st, 2012 at 8:44 pm) and I went to bed. I woke up hoping that this would all be fixed only to learn that my biggest client’s password that was finally working yesterday is now no longer working – as well as all but one of my other clients – and mine. So… what do we do Dreamhost???? Should we reset these passwords again? Or is that going to cause you more problems? Should we all just walk away from this for a day and hope that you’ll fix it? I understand that hacking happens and I feel your pain. This is really hurting you and I’ve always been a big fan. So I’m not blaming you for this mess. But please give all of us a better update on what’s happening and what we should be doing. Otherwise, you most likely have thousands of people resetting their passwords again today which is making the queue even worse. Please give us some direction.

People say “these things happen”. These things don’t happen to top tier hosting companies, because they know how to safeguard against it. Dreamhost should probably consider just reselling Amazon Web Services hosting, instead of using their own hardware and team, because AWS has their shit together.

I changed my mail passwords yesterday and the new mail passwords worked. Now they’ve reverted to the old passwords. How is that even possible? When I enter new passwords and they start working, how can there even be a record of the old passwords anywhere? My guess is that there’s been a restore operation we weren’t told about.

Adding to my previous post: I deliberately changed some other settings when I changed my mail passwords, and those changes have reverted too. This adds support to my theory that they did a secret restore from a day-old backup.

I have experiences similar to matt’s: I changed email passwords and the new password worked until they reverted back to the old. I changed them again, they worked and now they’re reverted again!?

I had the same experience (new password set y’day which worked and then reverted again today). I hope they provide an explanation for this odd behavior. It’s nice to get a confirmation if they restored the settings from an old configuration.

I managed to get one of my user accounts password changed and after a about 30 min I had FTP access to my site. However now I have permissions issues and can not upload a modified file. I have a support ticket in but have not heard a thing in about 3 hours. What is up dreamhost?? I gotta get stuff done…My Clients are getting upset as am I

ALL my files have been deleted and I can’t get any customer support. I tried to do a domain restore but nothing has happened for over an hour. PLEASE START PROVIDING CUSTOMER SUPPORT.

Confirming – my email password was secretly reverted as well. That’s not good behavior.

Oren – I’ve had Shell/FTP *and* email passwords revert in the last 12 hours as well (email has reverted to old passwords TWICE). I have flagged this with support incase it’s another security issue but they haven’t acknowledged it yet. :(

@ Herp Derp:

These things DO happen to other hosting companies all of the time. But most people here are on a “shared” web hosting account — not exactly a “top tier” arrangement. “Top Tier” would be more along the lines of your own server at Rackspace, and your own IT department managing it. Everything else is just vanilla pudding.

I get a kick out of how many people’s hair catches fire here when something goes down on a server they are probably paying no more than $30 a month on. Their “clients” are all having fits, and they are “losing thousands of dollars” because a piece of electronic equipment does what you can expect ALL electronic equipment to do from time-to-time — fail.

Want a cloud network Go to a cloud network. But don’t expect to pay the price you are paying at DreamHost.

But please people, quit trying to impress on the rest of us how much money you are losing because something went wrong with your server. If you really were relying on your website(s) for thousands of dollars worth of business every day, you certainly wouldn’t be spending $10 (or even $400) a month on a server at DreamHost. Sheese.

I fully understand the frustration in losing control of your website(s) when something goes down. Right now, I cannot access MY FTP/Shell because of this. But I also know that if I just wait a while, try it again, it will probably be working again. If not, I will put a support ticket in and they will handle it ASAP. Problem will be solved, and the end of the world will still not have occurred.

But in the mean time? I can pound my fist, look at other hosts, threaten to move ALL of my client’s websites to some mythical place where nothing like this happens. I could spend moments venting my anger at DreamHost and question their “sloppiness” or “unprofessionalism”. And for what? Most of you will continue to stay here and wait until the NEXT time and do the routine all over again.

Take a chill pill. Relax. Spend some time on Stumbleupon. Everything will work out, I promise.

See? In the time it took me to write that thing above, I can now access my FTP/Shell. Isn’t life grand?

Even with the occasional faults and foibles, I still prefer DreamHost over any other “discount” hosting service that I’ve tried.

DJ Allyn is right on the money – if you HAVE to have things NEARLY PERFECT, be prepared to spend LOTS more than you do with DH… and then stuff will happen anyway.

I just read Simon’s post ( http://blog.dreamhost.com/2012/01/21/security-update/ ) about this issue, and I’m completely satisfied that DH handled this very well. Also, from the description he gave, it could have been any hosting company and they would have been broken into the same way – a previously unknown exploit is… well, unknown!

What I don’t understand is why you are allowing plain old FTP at all: with that protocol passwords are transmitted in the clear!

My account was set up with that and I immediately changed it to sFTP/SSH only and disabled FTP. But I wondered (and now wonder again) why I would have to do that in the first place?

It’s very odd to have the *least* secure option as the default.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host homie-vserver198 moved to new hardware and restoring from backups (RESOLVED)
3:17 am

Web VPS host homie-vserver198 moved to new hardware and restoring from backups (RESOLVED)

Posted (January 21st, 2012 at 3:17 am PST) by jj

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver198′, we have moved the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver198’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 25th, 1:32pm PST: The restore is complete and all VPS guests are online! If you notice any issues with missing or old data on your account, please be sure to contact support so we can help! -Seohee

Update Jan 25th, 10:36am PST: Our Admins are working on fixing up 4 remaining VPS guests that are still offline at this time. All other services are online at this time! If you notice any issues with missing or old data on your account, please be sure to contact support so we can help! -Patrick

Update Jan 25th, 7:47am PST: A final round of restorations are still running. Unfortunately, a few of the VPS guests still remain offline while this continues. Our admins are continuing to work on bringing those back online and we will have further updates on that as soon as we have more information. If you notice any issues with missing or old data on your account, please be sure to contact support so we can help! – Chalise

Update Jan 25th, 5:24am PST: A final round of restorations are continuing. Unfortunately, a few of the VPS guests remain offline while this continues. Our admins are still working to bring those back online and we will have further updates on that as soon as we have more information. If you notice any issues with missing or old data on your account, please be sure to contact support so we can help! – Chalise

Update Jan 24th, 10:27pm PST: A final round of restorations is going on now. Unfortunately it appears a few of the VPS are offline due to this. Our admins are working to bring those back online. If you notice any issues with missing or old data on your account, please contact support and we will be happy to assist! -Dan J

Update Jan 24th, 2:41pm PST: We are very very nearly done with this server restore! Our Admins are doing some final checks before we mark this issue resolved. If you notice any issues with missing or old data on your account, please contact support and we will be happy to assist! -Seohee

Update Jan 24th, 11:43am PST: We are very nearly done with this server restore! Our Admins are doing some final checks before we mark this issue resolved. If you notice any issues with missing or old data on your account, please contact support and we will be happy to assist! -Patrick

Update Jan 24th, 7:17am PST: The restore process is running smoothly and all VPS guests remain stable and online. If you experience any issues with your VPS on this host machine, please make sure to contact support! – Chalise.

Update Jan 24th, 3:05am PST: All VPS guests should continue to be up and running normally on this host machine. The restore process is also running smoothly. If you experience any issues with your VPS on this host machine, please make sure to contact support. – Chalise.

Update Jan 23rd, 10:25pm PST: All VPS guests should be up and running on this host server at this time! If you are experiencing any issues with your VPS on this host, please contact Support and we will be happy to assist! -Dan J

Update Jan 23rd, 1:03pm PST: All VPS guests should be up and running on this host server at this time! The restore is approaching 80% complete! If you are experiencing any issues with your VPS on this host, please contact Support and we will be happy to assist! -Patrick

Update Jan 23rd, 10:04am PST: All VPS guests on this host should now be running. Those customers who have contacted us about their guests being offline should be hearing from us soon in regards to their specific situations. -Brandon.

Update Jan 23rd, 3:05am PST: Most VPS guests on this host should be running and restored by now; a few guests do appear to be offline and we are in contact with our admin team to get them back up and running as soon as possible. More information to come shortly. – Chalise.

Update Jan 22nd, 10:13pm PST: The restore process continues to move along without any hiccups. We’ll continue to post updates on the restore progress. -Seohee

Update Jan 22nd, 6:00pm PST: The restore process continues to move along without any hiccups. We’ll update this post again shortly with its status. -dan w

Update Jan 22nd, 2:30pm PST: This restore process continues to progress without issue. Sites should continue to respond during this process and we will continue to post further updates until the process is complete. -dan w

Update Jan 22nd, 9:00am PST: The restores on this server continue to run without issue. Our admins have fixed up any VPS guests that weren’t running on this machine, so everything hosted on this server should be working at this time. We’ll update this post again shortly with more info when we have it. -dan w

Update Jan 22nd, 5:30am PST: We are aware of trouble with some VPS guests on this host machine. The restore process continues and we are continuing to actively work on this issue. We will continue to post further information as it becomes available. -JJ G

Update Jan 22nd, 2:30am PST: Several VPS guests have already been restored to the new hardware and the restore process continues to execute properly. We’ll continue to update this post as the restore progresses. -JJ G

Update Jan 21st, 9:55pm PST: We are doing a full failover on this server. At this time all VPS servers have been mounted to a new server and are running from backups while the restore is running. The restore process seems to be working well. We’ll continue to update this post as the restore progresses. -Dan J

Update Jan 21st, 3:20pm PST: We are doing a full failover on this server. At this time all VPS servers have been mounted to a new server and are running from backups while the restore is running. We will continue to update this post as the restore continues. — Oscar

Update Jan 21st, 5:28am PST: The file system check is still going through the 1st pass and is 85.61% done with that pass. We’ll continue to post updates as the process runs. — JJ G

Update Jan 21st, 4:36am PST: The file system check is still going through the 1st pass and is 50.12% done with that pass. We’ll continue to post updates as the process runs. — JJ G

This entry was posted on Saturday, January 21st, 2012 at 3:17 am and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

15 Responses to “Web VPS host homie-vserver198 moved to new hardware and restoring from backups (RESOLVED)”

The following machines are affected:
ps40204 ps40276 ps40297 ps40315 ps40515 ps40754 ps40906 ps40910 ps40976 ps40978 ps40983 ps41147 ps41150 ps41403 ps42003 ps42838 ps43282 ps43491 ps43629 ps43928 ps44450 ps45692 ps45937 ps45964 ps46000 ps46007 ps46134 ps46144 ps46455 ps46467 ps46657 ps46852 ps46873 ps47013 ps47289 ps47297 ps47583 ps48308 ps48975 ps49080 ps49177 ps49762 ps49989 ps50379 ps55292 ps62776 ps62781 ps63084 ps63179 ps63295 ps64393 ps72701

I don’t know if this is related. But ALL of my sites are empty, without any files displaying only a favicon and FTP is not working and I cannot even upload the files! If you are trying to drive me away, you are doing a very good job the past 3 days.

any new updates regarding this system check, currently everything is still down and not working

All of my sites show only
favicon.gif 21-Jan-2012 03:50 0
favicon.ico 21-Jan-2012 03:50 0
open/
but my VPS is not one of the machines shown above. Everything down 3 days already.

Check the previous post for all the people talking about sites that were hacked months ago. Malicious .htaccess files and javascript were inserted into many sites, some using WordPress, some using other types of software. Maybe whomever hacked the sites sees that this is the end of the game, so he’s just deleting stuff outright? The trojan installed allows the hacker to remotely create, modify, and delete files. Did any of you use the Dreamhost one-click installers for anything on your site, and if so, did it do an upgrade for you recently?

It’s been 10 hours since all my sites and server is down, can you at list give estimates on how long will it take to fix this issue? there is no one to talk too !

I am the designer for this site, and I can’t get on the ftp site. Can you let me know when I will be able to.

Thank you.

“We’ll continue to post updates as the process runs.”
Oh really? The last update was HOURS ago and nothing since. I suspect a collapsing house of cards and a scramble to put ‘em back. Sure hope my files and database are secure and there. Though I backup both the DB and files, today was the day for that so I’m a week old for all of my sites.

Yeah, I could do it more frequently and having to rebuild all domains won’t put us out of business, but the hours we’ll have to invest in putting things back in order means I’m outta here (and probably more, depending on legal advice).

The hacker that took down some of DreamHost web site is a known Israeli hacker name Ofer Sadot. He was born in 1956 and live in a city called Hod-Hasharon (in Israel).

His IP address is 192.116.227.130 .
He owns a web site for anonymous email sending and spamming (and some Porno and Pedophile, too):
http://www.noipmail.com

This hacker is using some proxies, but he can not hide for ever.
Some of the HTTP headers exposed him by his real home computer IP address (Cdn-Src-Ip, X-Forward-For, X-Real-IP, X-Client-Ip).

The Hacker Ofer Sadot ISP is “012Smile Telecomm”:
http://www.012.net.il

I used to call 012Smile and complain about this hacker, but they did nothing. The Israeli Police arressed him sometimes, but he is keeping to hack and spam web sites.

The past week he tried very hard, with some infected computers, to hack DreamHost servers.

i guess the file system check went wrong.. 6 hours without updates and the files are still in read-only mode.. come on.. we need to work!

I understand you’re dealing with many things that have gone wrong recently (so am I), but I wanted to chime in with the growing chorus of frustrated customers. My VPS is among those affected and all my sites have been down since last night. This is nearing a 12+ hour outage. I don’t even have read-only access; I simply cannot do so much as ping my server. As if it needed saying again, and I know it doesn’t, but please fix this ASAP.

Meitar — My sites have been down over 18 hours. Two tickets from today, one from 9:44am PST and another nearly TWELVE HOURS LATER at 9:34pm PST have gone unanswered.

Tweets to the CEO, DreamHostCare and DreamHost accounts have all gone unanswered.

It is after midnight now in Minneapolis and when I awake in 6-7 hours, if my >$75/month VPS isn’t up and running I’ll take all of my backups and setup another host and switch DNS tomorrow.

My ticket to sexual pleasure has gone unanswered.

It seems every one of you is tied to that tiresome old notion of two distinct genders.

For those of us who are both and neither, come on down to the wild side.

Punch my trouble ticket. Go down on my server. Link up to my uptime.

I bet you’ll never care about those troublesome web hosts again.

All sites are back up but now have permission errors? This needs to be fixed!

My VPS has been down for days now. Don’t you think I deserve a little more from your support?

I have a Google Ad Words campaign linked to a website on this server. My helpdesk application, used by all my customers, is hosted there as well. I’d never let my customers in the dark for so long in such a serious matter.

2012/1/23 André Almeida
Any news???

2012/1/23 André Almeida
Update Jan 23rd, 1:03pm PST: All VPS guests should be up and running on this host server at this time! The restore is approaching 80% complete! If you are experiencing any issues with your VPS on this host, please contact Support and we will be happy to assist! -Patrick

ps63295 is still damaged!

2012/1/23 André Almeida
Don’t you guys think this is taking too long?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Changing Shell/FTP Passwords due to Security Issue (RESOLVED)
9:25 am

Changing Shell/FTP Passwords due to Security Issue (RESOLVED)

Posted (January 20th, 2012 at 9:25 am PST) by Oscar

Last night we detected some unauthorized activity within one of our databases. While we don’t have evidence that customer passwords were taken at this time, we’re forcing a change out of caution. Please login to our web panel and change any passwords you may have with us. We’ll keep this post updated as we get more information.

Update Jan 23rd, 12:16pm PST: We are still actively working through hundreds of support requests due to the delay with password changes in the Web Panel. The great news is that password changes in the Web Panel are now taking less than 10 minutes to complete!

All of our mandatory password changes were done over the weekend, so if you are unable to log in to your FTP/SFTP/Shell accounts at this time, please request another password change and that should resolve your issue!

If you still can’t access your FTP/Shell after that, please contact support with the details, so we can assist you. -Patrick

Update Jan 22nd, 1:05pm PST:Our support team has handled thousands of requests over the weekend to help customers through the process of selecting a new password. All mandatory FTP/shell access password resets by DreamHost were completed Friday evening for our shared customers and by Saturday for our VPS customers, so if you’ve changed your passwords since then, those should be active. Any issues, please continue to message support through the panel. Our average response time is a bit higher than normal because of the number of requests, but we have a large number of our tech support team working diligently to respond to your issues. Due to the fast action we took to reset passwords, we’re not seeing any unusual malicious activity on customer accounts. Our security software and systems are functioning normally.

If your password isn’t still working, please try another reset, and try accessing again in about half an hour. Currently, that’s how long it takes for the new password to start working on your web server, and this time is decreasing as less customers are requesting resets. Please note, we have a newer status post up about the password reset delays, so you will see more current updates there.

If you still can’t access your FTP/Shell after that, please contact support with the details, so we can assist you. -Andrea

Update Jan 20th, 09:50pm PST: We’re going to set this post to resolved. We have been sending out update emails to every account owner we have, letting them know what happened, and how to proceed from here on out. As a precaution, we advise every user to change all email passwords as well. We are not forcing this change, however, so make sure you take care of that ASAP. — Brian H

Update Jan 20th, 08:15pm PST: Forced updates to FTP/shell users should now be done. Any users that are locked out should update their passwords via the web panel. If you’re still unable to get in, please let support know asap. — Brian H

Update Jan 20th, 04:00pm PST: We’re still updating all users to force a password change. We should be done with this soon. Also, you can expect an email with more details to follow. It does take some time to email all of our customers, so please keep that in mind. — Brian H

Update Jan 20th, 02:35pm PST: I apologize, my previous update was incorrect. If your password has been updated by us, your old password will not work. You must login to the web panel, and change the password before you can login via FTP/shell user again. These updates are taking a little longer than normal, as announced previously. We apologize for the delay. — Brian H

Update Jan 20th, 01:57pm PST: We’re still pushing through all password updates on our system. Passwords should still work on your account. If you’re having problems, please submit a support request and our staff will get in touch with you ASAP. — Brian H

Update Jan 20th, 12:57pm PST: Password updates are still humming along. In our preliminary checks, it does not look like any customer accounts where accessed. — Brian H

Update Jan 20th, 12:40pm PST: Password changes are still going through, so some of you may still see old passwords working. We’re doing everything we can to make sure all passwords are updates ASAP. — Brian H

Update Jan 20th, 12:15pm PST: Panel access is looking much better now. Passwords are being updated, but due to the influx of requests, updates to passwords might take a little longer than normal. Our servers are working overtime to keep up with demand. — Brian H

Update Jan 20th, 11:55am PST: Web panel slowly returning to normal. We are still working on getting it 100% restored, so be aware you may see some downtime there, for the time being. — Brian H

Update Jan 20th, 11:32am PST: We’re still working on getting the panel back up and running and apologize for the delay. — Brian H

Update Jan 20th, 11:12am PST: Currently, our web panel is a bit overwhelmed and is showing intermittent access. We’re working on fixing this issue, so passwords can be updated. — Brian H

Update Jan 20th, 10:59am PST: We apologize for any confusion as to what the current procedure needs to be. We recommend you change your password for any FTP/shell user you have with us, right now.
If you’ve already changed your password this morning, thank you for being so quick! We are still running through all FTP/shell users, just in case, so it’s possible we’ll change it out from under you again. We apologize for this duplication of effort, but we’re trying to get things secure as quickly as possible. — Brian H

Update Jan 20th, 10:20am PST: The process for changing all passwords is still going. After we complete the password changes, all FTP/shell passwords MUST be changed by you in the panel.

To edit your password in the panel, please log into the web panel and go to Manage Users. Click edit next to the FTP/shell user on the right and you can change your password there. — Oscar P

Update Jan 20th, 10:00am PST: We have started the process of changing all Shell/FTP passwords now. After we complete the password changes, all FTP/shell user passwords MUST be changed by you in the panel.

To edit your password in the panel, please log into the web panel and go to Manage Users. Click edit next to the FTP/shell user on the right and you can change your password there.

Update Jan 20th, 9:45am PST: We are doing this for all customers and we will be changing all Shell/FTP user passwords.

This entry was posted on Friday, January 20th, 2012 at 9:25 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

647 Responses to “Changing Shell/FTP Passwords due to Security Issue (RESOLVED)”

Will this relate to all customers, or just a subset of customers that were stored on this db?

ALL FTP/Shell accounts on ALL servers?

Can we get a little more clarification on this? Are all shell account passwords being changed? On every dreamhost server? What does this mean for customers? What steps, if any, do customers need to take? Can you tell us what you detected last night in one of your databases last night?

My account was compromised 2 days ago. Htaccess modified and malware on all my WordPress sites glad to see I’m not the only one effected.

@Chad, we’re still looking into what may have happened, so we don’t have those answers just yet. It’s best if you just login to the panel, and update all of your passwords with us, better safe than sorry.

Well, I agree with Chad, this is not a “small” issue… Are you sending a mass email to your customers to tell them about the breach?
This is a really bad week for Dreamhost!!! (And Jazz server is still down every now and then)

I had a number of my sites compromised about two months ago. The root cause was some malicious scripts that got inserted into the .htaccess files of my Drupal, WordPress and just everything else I have out there.

At the time, DreamHost had nothing to say about this other than it was probably something I did other than any security holes on their shared servers.

I finally got the mess cleaned up and set all my .htaccess files to read-only. Haven’t had a re-occurrence of problem so far.

I’ve been running some cron scripts since then that check for the particular malware that infected my domains.

Wait, are YOU going to change my passwords, or am I supposed to log into the Panel and do it myself? Your first update to the post makes it sound like you’ll be updating my passwords for me.

does this apply to all other types of passwords like SVN repos? or just dreamhost usernames?

This looks like a recipe for disaster. The post is very unclear. If DH is changing the passwords, how will we know when you’re done? I don’t have time to check over and over again to see if the shell passwords have changed. I foresee many angry DH customers who can’t get into FTP or shell accounts because they didn’t know about this.

@Ben, yes we will be email all customers regarding this.

@Ken: Yes, we are changing your password right now. After this is done, you can then change your password again to something you wish. For right now, hold off changing your passwords until we’re done changing it.

@Chrisw: Right now it only applies to shell/ftp passwords, but you can change all passwords just to be safe.

@basscrest we’ll let you know when we’ve changed everything. We’re also posting a message right in the middle of our web panel, once the change has been made. We’d rather be safe, than sorry in this situation.

How could they possibly have accessed customer passwords? Are you storing customer passwords in plaintext??

In this context, is shell/ftp synonymous with ssh/sftp? I’ve been careful to only enable encrypted logins on my accounts. Are these passwords affected as well?

I’m looking forward to reading the post-mortem on this one.

I’ve had several sites’ htaccess files modified with redirections, plus some .php files appearing out of nowhere.
I don’t feel safe on Dreamhost anymore.

@Brian – but it’s still confusing. There’s a message on the Manage Users page (I happened to be there for a different reason than this) which reads:

Due to some unauthorized activity we detected within one of our databases, we have forced a reset of all FTP/SFTP and shell passwords as a precaution.

Please visit the “Users > Manage Users” tab immediately to change any passwords you may have with us.

So I changed all of my passwords right away – because it says “immediately”. But you’re saying here that we should wait until this process finishes. So now I’m concerned that the process is going to come through and change the passwords that I just changed. I have to let the people who use these passwords know what they are – I just don’t know when to do that.

FINALLY, something is being done about this. I’ve been tackling a spam problem where someone has been accessing my servers and inserting php files (example: commone.php & editing style.php) which lead google to see a bunch of spam results. They also insert something called “coockies.txt” and modify the .htaccess files. Whenever I delete them they reappear approximately ten minutes later. Ridiculious that this could be happening across the entire network.

I can’t access the edit function – super frustrating, as i’m in the process of setting up client sites.

Clarity as to what DB was compromised, if not all of them, and what security measures are being taken in regards to sever maintenance on DH behalf. I manage many Drupal sites, so at this point I am looking at possibly telling all clients they may have to reset there account information on our Drupal sites, because the DH DB’s were breached.

Anxiously awaiting on news as to how this happened, and what is being done to precent any future issues that may arise.

In addition, in recent months, I also have been having issue’s with Google flagging my sites as being redirects and storing malicious content / code.

Let’s get down to the bottom of this.

This is creepy to me because I have had an ongoing “hunch” that the next phony false-flag psy-op on the American sheople will be a new dastardly “cyber attack” riddle, bundled in a “cyber heist” mystery, wrapped in an “internet shut-down” deception …

OK, that last update clarifies things. I won’t tell my users the new passwords until I’m sure that the process has completed and I can make sure that I know they’ll work.

Also, the panel seems to be overwhelmed at the moment – getting “connection has been reset by the server” messages on a consistent basis. Guess I’ll wait until this is resolved to do my changes.

@basscrest Yea, I just posted an update about that. In the interest of being as secure as possible, you may have to update twice. I know this is a pain, but its much better to err on the side of caution.

@Kevin FTP/Shell user means ssh/sftp, yes.

@Hector htaccess modification is something different. That is almost always related to outdated, insecure software you’ve installed on your account. This is not related to what’s happened today.

the webpanel seems to be down?? :/

@Brian – thanks. I’m not opposed to having to make two changes, just wanted to make sure that I could communicate down the chain effectively.

And the Panel is down. Good Job DH! LULZ

Yup, my web panel is down as well. Just as I was going to edit some critical DNS info. Damn universe.

As of 11:10 AM, the control panel is DOWN. “This webpage is not available The connection to panel.dreamhost.com was interrupted.” I was 1/3 of the way through changing my passwords and it locked up on me.

I can’t even access the panel to change anything.

Why is the panel down and when it will be up again?

I’m really frustrated with the webpanel being down. I’m trying to get a domain up, I won’t forget this bad experience.

Yeah, four days of dorbo attempting to come back on line, they can’t tell me if I’ve lost all my non sql data from Jan 7-16 (which I’ll have to attempt to recreate) because “backups are still ongoing” — which are now just at 70%, and I can’t get into the webpanel to change PWs… as well as being victim to ALL my sites having their htacess and malware PHP files modified/loaded regardless of if they were running the LATEST WP (per one-click install, set to auto-update) etc. multiple times in the past 5 months. (To their credit, they were able to run a script and clear out the crap, but still… I never got the email about that, and it took my looking at “support history” to find out they sent an email that was never received by me alerting me to it (and yes, I check my spam filter, etc. daily.)
I’ve been a dues-paying fan for seven years, and I realize you get what you pay for, but it seems to all be coming home to roost RIGHT NOW. A shame.

I too had Google flagging my sites as storing malicious content / code. See my previous post. I suspect this problem has been around for months. It hit me in three different domains and all their subdomains. Each and every .htaccess file in all of my domains/subdomains had been infected with a malicious script.

I do believe the flood of people changing their passwords has caused the Webpanel to crash.

Care to send us an email when it’s working again? =/ I’d like to change my password and pay my bill before my site gets taken down.

@Brian Maybe, but still it’s something that had never happened to me before, and I’ve been on Dreamhost for more than a year now.
It has never been my fault that the htaccess in the sites I admin were changed for something else. I use mostly WordPress and I always keep it as updated as possible, plus, I got to repeat: it’s been happening for weeks now. Just today I had to delete one .htaccess file on a site that only had files like .jpg, .html and .css.
I just want to feel that my sites are safe on DH’s servers, that I won’t lose traffic or get bad points on SEO because of something that clearly is not my fault.

@Brian “htaccess modification is something different. That is almost always related to outdated, insecure software you’ve installed on your account.”

All I have installed on my account is a fully up to date phpbb forum and a small textpattern blog, also up to date according to the panel. How is the .htaccess still happening? It literally changes the minute after I delete everything.

Weird webmail behavior on some (but not all) accounts. And now also can’t access panel.

So Brian’s reply at January 20th, 2012 at 11:08am indicates that hashed passwords were accessed. Brian, can you clarify if you’re salting the passwords? Were the salts compromised as well?

I’m also curious to know how much data was compromised. Is my credit card data safe?

The Control Panel is down, can’t access.

@Ken I didn’t indicate that. I said we store passwords that are hashed. We noticed unauthorized access to one of our databases. Any time that happens, we have to take precautionary measures, and that’s exactly what we’re doing. We’re still investigating what exactly was accessed within that database. Credit card data is stored on a completely separate system, and was not accessed, nor were web panel logins.

Control Panel access is back, but if you try to edit any FTP passwords, the page just hangs.

damn! that’s a small post for such a major issue imo. think of how many passwords that would be… curious to know… (hehe…)

I’ve had the same problem as everyone else here, with the pharma hack / bad google search results. I can’t get into the webpanel, so assuming like TC says above me that it’s crashed. One of my websites http://nationalshows2.com is down (same one that had the pharma hack), all others are up.

As opposed to Mike’s comment that the Panel is back, I still cannot access

@Keith Right after I posted it, the Control Panel died again. I can’t access it either.

The panel’s down again for me…even after clearing cache. This is the second day where the control panel has had a problem loading the page. (Error Message: The connection was reset The connection to the server was reset while the page was loading.)

Blarrrrrrgh!

The number of times you absolutely need to access your control panel seems to be in direct correlation to the number of times it is down.

And here I was going to log in and pay my DH bill today. I guess they will have to wait until NEXT month, which will be the next time I remember to access my panel…

Toodles.

You can always log into your shell account and use the passwd command.

Also not able to access the panel. Amongst this catastrophic event, it appears that I am losing a whole days worth of work over this.

i had(ve) the pharma hack on a weekly basis – even did a plain jane WP install with no plugins and standard hardening. Still hacked. Came the point that other users on the system was writing files to my webroot. emails to dh support came back with standard template replies and ignored the actual questions. (even sent them a screenshot with the user – they had no response to why or reassurance it would stop). Happened again last week.

Customer for 8 years and I’d say the last 6 months the security has become lax (and the customer service went from prompt personal replies and explanations to standard templates 2 days after the fact)

I was hacked on almost all (maybe all for what i know) WordPress sites. I got a message from DH taking down one site that was phishing.

I removed and reinstalled wordpress on most. some of the hack pages came back up. I don’t know exactly how to remove all of the infected files while keeping my content and other important information. I don’t even know where to look for all the infected files. I have over 20 infected sites (that i know of).

Now, there have been some emails sent out from my admin on some of these WP sites.

DH needs to step up, define the problem and let us know what to change and fix. Other than just the FTP Shell. And even if we change the password how does this affect the already infected sites?

About infected sites, I’ve found some malicious code in the header of, at least, one of the sites that got its htaccess changed. It was a WordPress and the code was inserted in all PHP files (including themes and plugins). What do I do about it? What if all the WordPress sites are infected like this case? As I’ve been reading the comments in this post, I think that all the WordPress I manage are already infected. I don’t really picture myself editing all the PHP files of every site (over 20) to delete the malicious code.

The same here!
And they told me it was my fault…

> i had(ve) the pharma hack on a weekly basis – even did a plain jane WP install with no plugins and standard hardening. Still hacked. Came the point that other users on the system was writing files to my webroot. emails to dh support came back with standard template replies and ignored the actual questions. (even sent them a screenshot with the user – they had no response to why or reassurance it would stop). Happened again last week.
Customer for 8 years and I’d say the last 6 months the security has become lax (and the customer service went from prompt personal replies and explanations to standard templates 2 days after the fact)

Hate to be a punk:::::: But this stuff always happens on freaking FRIDAY’S I got soooooooo much work to do damn. I cannot login to my shell, cause I changed the password and it is taking forever to reset!!

=(

To put this all in perspective, Dreamhost is not alone in this mess. This week is one of the most significant hacking weeks in many years. “Hacktivism” is at an all time high – and we are paying the price. Anonymous, et al, care about big results, not about our individual livelihoods – a paradox at best.

While I know very little about that world, I wouldn’t doubt that DH’s problems are potentially connected. And if the FBI can’t prevent it on their own sites, it’s understandable that Dreamhost would be vulnerable as well?

Let’s take a deep breath, and as @Vince T says: Please let us know what we can do as individual DH customers to protect our servers and this in the future?

jake

Control Panel is back, just finished changing all FTP passwords, now it’s time to see what kind of damage was done. This took an hour, and on the last two hack attacks I had to go in and scrub code on multiple sites, which took another three hours. This is the third incident of this kind in less than a year. The time has come for DreamHost to realize the hacker community has painted a big, bright bulls eye on their backs, and they need to do much more to protect their customers. This is unacceptable.

Hello,

This is sad news. Every other week we are having some kind of downtime.

Brian or Oscar (or Dreamhost for that matter), did you change database users passwords too? I’m having a few problems with authentication there too…

So this can explain the .htaccess hack I’ve suffered on all my websites?

THIS is why I switched hosts to HostGator last week. Dreamhost reliability has been so shaky for the last year or two. I got fed up with my site going down that I switched hosting companies. I had several conversations with support and it was ALWAYS some script I was running on my site. One week in with HostGator and my site is faster and no down time.

BYE BYE Dreamhost!

You notice the trend? All these people claiming they’ve been hacked are using WordPress.

I noticed a script running against my WordPress install a couple weeks ago. Something like 200 requests were being made to the server. They were searching for vulnerabilities. I banned the IP, and installed the proper plugins to prevent it from happening again.

Don’t blame DreamHost if you’re too inept to protect your own files. It isn’t their fault such vulnerabilities exist in WordPress, it isn’t their fault you don’t know what you’re doing, and there isn’t anything they can do about either problem.

How can we check if WP installations and .htaccess have been compromised?

That cause me allot of problem …..allot of china spam come on my site ? and i don`t know how i do f__g reading of them please help ……..

I change my password but it doesn’t refresh arrrghhh

I don’t know if this is at all related, but I was installing ZenCart to one of my domains yesterday and upon doing the database setup I was receiving errors regarding a domain I don’t even own (etymolution.com). Still not sure if it was because of an error on my part, or if someone got to the database before I did, but I had to wipe it and start again.

Que ocurre tengo muchas

so has DH staff finished changing all shell/ftp user passwords?

I am testing a few of my shell users and see that they are able to login with their current password so obviously I’m thinking DH has not changed all passwords yet – correct?

I had my site compromised , vbulletin , the offender inserted a base 64 crap into just about every php file , as a result my site is no longer working

have cleaned up the base64 stuff , but clicking on a thread leads to a blank page

@John re: You notice the trend? All these people claiming they’ve been hacked are using WordPress.

Sure – My htaccess file is about 80k with banned IPs from countries to tor exit nodes. Limited access from POSTS, to wp-admin/wp-login from my own IP, banned bots, browsers, etc. I view my log files every day (error.log and see mod_security is working). The problem still happens when a user is able to gain access to the filesystem and upload their backdoors. Hard to complain for shared hosting costs but at least dh support needs to answer questions to why it still happens. In fact – I would love to know if security hole exists in WP so it can be patched back to the community.

Does this apply to all passwords, including email and database passwords? Or just users?

My passwords were all reset and Im still not getting access to my FTP.

I understand that there is an issue with hacking overall, but there really needs to be more done to protect your customers from this kind of stuff. I’ve been dealing with hackings to multiple sites of mine for quite some time, and it’s really getting kind of ridiculous now.

I disagree that malware injected into .htaccess is “always” due to users running outdated application software. Too coincidental for this many different DreamHost users to all have similar problems.

But wait! Maybe I’m wrong! If I consider the version of Linux DreamHost uses and their diligence in applying security-related patches to all the “applications” that comprise the GNU/Linux operating environment.

I won’t feel secure until I see some evidence that the ROOT CAUSE of these problems has been identified. I’ve been a DreamHost user since 2000-12-30 and this last year or so I’ve had more problems than the previous 10 years combined. And I seriously doubt it’s because of my screwups.

@nikos same same – didn’t noticed any changed passwords. Luckily was able to change them via the panel. All done now. Guess wouldn’t heart to change the database passwords as well.

@John I also said it has happened to sites that were not WordPress, at least in my case. It isn’t an only-WordPress problem. There went your logic.

People WordPress is a shit system, stop using and code your own stuff. When a platform is hot, it will be a target for hacks!

What concerns me the most is that I never received any email or text message notification that there was a potential compromise of one or more of my websites. Nothing. If I had not logged into DH cPanel on a routine issue I never would have known. This is inexcusable.

I think that instead of sending out really laid back dude hippy-dippy super cool monthly “newsletters” (which are invariably late), DH needs to wake up and start acting like the professional internet service provider it aspires to be. I am sure the engineers and managers at DH are truly talented and dedicated people, but the perception given by offerings like its newsletter and advertising, coupled with a serious security lapse such as this, make DH and all the people who work for them sound like some laid back dude hippy-dippy super cool organization that can’t get its act together. This is not the kind of company I want provisioning and managing my critical infrastructure.

I had hoped that the new management would have started down the path of increased professionalism. I was wrong. I hope *this* wake-up call does not go similarly unheeded.

@Chad Buie – that’s about as smart as saying that people shouldn’t use webhosts and run their own servers. After all, then you’d be able to handle the attacks, outages, etc. ASAP since you’d have the server in your house. Most people don’t code. After all, that’s why there’s Word Processors, Spreadsheets, and website editors.

Gotta love it when someone makes a generalization and then says “oh, do it yourself.”

I have changed my passwords but can not login to Filezille still?

I changed my ftp passwords in the Admin panel and still am unable to login to any of my sites. Anyone else having this issue?

I’m not going to be about all day tomorrow, I am moving houses and this is the last thing I needed.

Please can you explain why I’m unable to login to the Filezille and I need you to make sure you secure

my passwords as I will not be around!

yup… i’ve changed all my passwords and am still unable to login

I think DH needs to answer here. It’s embarrassing to be phoned by 3 clients about the problem and I lost one because of this. Seriously considering not to renew.

Wonder if http://anonyops.com/ had anything to do with it, that hacking GROUP!??

Web-based FTP via the Control Panel is working. FTP programs like Filezilla are not. I’m getting to the point where I may keep my domain registrations with DreamHost and begin to transition the hosting elsewhere.

@george, I just tested your login, and it works correctly. Are you using the new password you changed to? For anyone having problems, please submit a support request, and we can look into it. Also, please remember, that, if you change your password, it takes a little more time to update than normal. All passwords I’ve spot checked, do work, however. Please submit a support request with the info you’re using and we’ll check it. Make sure you let us know if you’re using your old, or new password (don’t paste the actual password, however)

Its not a username issue, the FTP site isn’t even lettings us connect. I don’t even get the welcome message (never even get to the point where I authenticate). It dumped me right in the middle of uploading some files about an hour ago.

For the people mentioning it’s only WordPress that’s affected by the .htaccess hack its not. I do no use wordpress on either of my sites. I use phpbb forum (which is up to date) on one and TextPattern CMS (also up to date) on the other. The hacking/modifications are only affecting the phobb installation.

This is super stupid. And weak. You need to hire better sysadmins.

WordPress sites on other hosts have been hacked recently as well, with what seems like similar measures.

Regardless of the security issue, there will always be loopholes in popular scripts as said above. When it’s secure, it’s up to you to keep your site and databases safe.

I had a bot try to send over 830 links through a personal URL shortner I have trying to hijack it.

Stay safe, companies are getting hacked left and right.

Did you put anything in place to stop people from setting the same passwords as the ‘possibly compromised’ ones? NO.
Did you tell people about this by email, or using any method other than them having SSH stop working? NO.

Amateur hour.

I’ve changed FTP passwords and can’t connect. It’s been an hour since the last post here, is this still a problem for others?

It’s not just FTP either, the webftp isnt working right either. I cannot upload files. It has been trying to upload the same file for 10 minutes now (a 10kb html file)

@James I have tried to change my password, but it just shows a little clock thingy, so they haven’t let my change go through. I have a .gpg file that has critical information I need access to on the dreamhost server i rent space on. This IS downtime for me, forget the websites I have I really don’t care.

I will be remembering this when it comes time to possibly restore. REALLY AMATEUR DREAMHOST!!!

@Steve, I completely agree about the level of professionalism. I too had those hopes when they got a new CEO. Then the new guy goes ahead and participates in the hippy-dippy newsletter! Oh well.

I still have frequent very-temporary e-mail authentication problems almost every day, network outages happen way too frequently with poor customer feedback (why isn’t this stuff all “highly-available?”), and then now I’m totally confused about what happened with passwords and whether or not my passwords are getting changed for me or not and what I should do NOW versus “in a little while.” Had I not been reading this external site in my RSS reader, I would not have known anything was going on either. I certainly don’t use the control panel on a daily (or even monthly) basis.

I’ve been a customer for many years and generally happy — what you get for the money with these guys is really pretty decent. But since I use my domains almost exclusively for e-mail, it might just be time to switch to gmail.

the password of access the control panel is safe?

;-)

HELLO DREAMHOST WHEN ARE YOU GOING TO ALLOW MY NEW PASSWORD TO TAKE EFFECT?!?!??!!

Hello Dreamhost,

What are you thinking? Do you want to get everyone ticked off at you, or just figure that nobody cares about the things they’re storing on those servers that they’re paying for?

Your SLA is a joke. I think you might refund me under a dollar for this. But it has done way more than a dollar worth of damage to me.

It was anon, I know because they’re all over the media right now! Burn the hackers, burn them I say!!

Finally able to connect after changing my FTP password. Took awhile, but i’m in now.

I believe this is at least partially due to what I call the ‘second shift effect’.. A ‘second shift’, which is usually on duty when many people are on vacation, or Friday afternoons, tends to ‘cowboy’ more — shoot first ask questions last. They either have ‘something to prove’ to the first stringers, or simply don’t know better… This is true with pretty much every company I’ve ever dealt with.

So, Dreamhost…. Who exactly approved locking me out of my account that I pay for? How much discussion was given to that decision? Why was I not notified in any way? How can I trust you to not do this kind of BS again?

“Update Jan 20th, 01:57pm PST: We’re still pushing through all password updates on our system. Passwords should still work on your account. If you’re having problems, please submit a support request and our staff will get in touch with you ASAP. — Brian H”

Who is ‘your’??? Because I am not the ‘your’ to which you are referring. Ooh a support request? You mean me trying to change my password through your web interface isn’t significantly annoying? I am in an airport, Brian H. I am tired. Fix my stuff, don’t break it again, and give me money for my inconvenience.

Password change is working now. Must take a little while to reset.

@CJ Rivas

Not being funny, but you are a big girls blouse aren’t you. DreamHost detected there was something amiss and acted extremely swiftly to patch this issue. If DreamHost had sent around an email asking you to change your passwords, it could have taken hours, days or even weeks before you had acted. The way that DreamHost handled the situation was heavy handed, but appropriate. So what I am trying to say is; calm down, everything will be fine.

Thank you DreamHost guys + ladies, and keep up the good work :)

@Chris

No I am not. What did Dreamhost detect? I am a sysadmin, I train sysadmins. What was amiss? They gave me no notice of a problem, I am not saying they should have emailed me asking to change my password, but they could have emailed me telling me that my password had been changed. Not doing that is simply unacceptable.

Heavy handed but appropriate? Are you George W Bush?

Everything will not be fine. Confidence is a fragile thing, and Dreamhost have lost my confidence. I know what good hosting is and what it isn’t. Frankly, I’d question your sanity just for the “heavy handed but appropriate” comment, were it not for the fact that it’s very likely that you are not a real person but an account used by a Dreamhost employee to make it seem like they’re not quite as stupid as they actually are.

Kinda shitty that I just found out. An email this morning may have taken hours to get to me, but it would have gotten to me before I read this on ycombinator. Anyway thanks for the update ive reset my password.

sofa king we todd did

I’m not sure I believe that passwords aren’t stored in plain text. Why does the panel list my passwords for database users? Why does the panel tell me what I just changed the password to when I change a user’s password? Passwords should NEVER be displayed and should be hashed immediately.

@Mike, quite right. Pretty sure there’s lots of nastiness behind the scenes.. A compromised database wouldn’t expose passwords unless they weren’t hashed/encrypted. How is that allowed? Pretty sure if Dreamhost had any PCI stuffs to adhere to that wouldnt fly. So how does Dreamhost justify treating my authentication information so carelessly? Oh yeah! They’re assuming I don’t know rat sh*t from rice krispies!

@CJ Rivas, open up a support request if you haven’t already. I had the same problem as you and they responded to my support request within minutes and voila, passwords changed

I am concerned, does this mean that the passwords are not stored with a one-way hash? if this is the case I am seriously unimpressed as this is the only way passwords should be stored…

The web panel is temporarily unavailable while we perform maintenance on our database.
[Sources: billing]

Shoot me now!

i can’t even log into the cpanel…

“Users > Manage Users” tab Temporarily Unavailable gad dammit!

Grumble!

No access to editing my password and the page to submit a support request is unavailable.

For years I have wondered about Dreamhost’s apparent disregard for password security. They e-mail passwords in cleartext, they display them on web pages. I don’t understand this at all.

so we should really change all our pwords or what? whats going????

EXTREMELY FRUSTRATED. VERY BAD TIMING. ALL CAPS WARRANTED FOR THIS EXTREMELY STUPID COURSE OF ACTION BY DREAMHOST WITH NO NOTIFICATION!!!!

There has to be an excelent reason why they had the passwords stored in plain text instead of as a hash…

WTF? SERIOUSLY? I mean, I go in, I change my fricking password as I was instructed to, and it took. So I go about doing my work and you not only change the password again, you take down the control panel?

DREAMHOST. WAKE UP. This crap is COMPLETELY unprofessional, COMPLETELY irresponsible, COMPLETELY unsatisfactory.

Not to mention all the RAID issues you’ve had this week that CONTINUE to have to be restored, but may or may NOT provide the data that may have been lost.

I realize it’s shared hosting, but unless you pull a 180, and quickly, I’m no longer going to be able to even remotely recommend you when someone asks who I suggest as a host, for even their own personal site.

I’m glad to see I’m not the only confused and frustrated by this whole thing.
I’ve read that I should change my passwords immediately. Then I read that I should wait until Dreamhost has changed them – then change again? (and just when might I know that this had been done – I wouldn’t have even known about this issue if I hadn’t been on Google Reader reading a Lifehacker post – NOT where I would expect to get first hand knowledge of the problem)

First off – can I get a clear answer as to what it is *I* should be doing at this point. Change password? Wait?
I have some 50 accounts and I’d like to get this done as quickly as possible.

I did change one client’s account FTP password and now it isn’t working.

There is no *excellent* reason for storing passwords in plain text. My guess is that either 1) early on they found they were constantly getting password reset requests and found it easier to simply store passwords in plain text so they could tell users what their passwords were, or 2) they had distributed systems and it was easier to implement authentication using clear text passwords than it was to implement secure authentication mechanisms.

@CJ Rivas

I’m a member of DreamHost’s team because I disagree with you? Like other people have said, open a ticket and ask for help. If you are a sys admin, surely you know that security has to be treated very seriously. Yes I agree with you when you say that DreamHost should have emailed us to notify us.

What does George Bush have to do with anything, being British your comparison is lost on me.

Question my sanity for a heavy handed approach to security, I would rather have an over cautious host than an under cautious one. I do not work for Dreamhost, I don’t even live in the US. As I said before, calm down.

What the hell, I changed my passwords and they’ve been cleared out again? Like, this is the penalty for paying too close attention and jumping into changing my passwords.

I also changed my panel password. I hope to hell you guys haven’t fucked that up because I can’t call you on the phone without a lot of hassle (deaf).

Gentlemen and ladies, let us not fight amongst ourselves. There is a critical issue here that is far more important than whether DH reset our passwords. I believe the big issue is DH’s practice of storing passwords in the clear. I call on you to focus your energies on that issue. What I have done is go to the Contact Us page (http://dreamhost.com/contact/) and submitted a request to the Public Relations department. This is what I said:

————————–

During this on-going company-wide password reset issue, there has been a lot of discussion on the dreamhost status blog

http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/comment-page-3/#comment-265087

about Dreamhost’s apparent practice of storing unencrypted passwords. I think it’s appropriate for the PR department if not a top-level corporate executive, to make a formal statement regarding this policy. Insecure storage of passwords is a huge security risk and has now put a burden on what I assume is thousands of sysadmins and other owners/operators of domains hosted at Dreamhost.

Sincerely …

I am really mad at this situation, you cannot imagine how much time I’ve spent trying to fix all compromised websites that had .htaccess problems. Im really frustrated at this point and seriously considering going back to mediatemple.net

so do we have to change DB pwrods as well or just FTP/SHELL???

UGGGHHHH!

Ok, I login earlier and see the lovely “Change your password” notice at the top of the control panel. Ok, so I do it. Then, I put in a test redirect for the site. Simple. Easy peezy. Right? Nope, it has an aneurysm. So, I wait an hour for it to propagate (giving the benefit of the doubt). No luck. So, I kill the redirect. The site has a stroke this time. It’s been ultra super gone for hours now. Then, after submitting a ticket which hasn’t been answered, the control panel dies. It finally comes back, the site is still dead for no good reason (though I have been able to get in with Filezilla all along) and I am completely frustrated. What the heck?

3 or more of my website have been put on google’s malware warning, are you going to help me fix that?!!!

What a mess. Just say “change your passwords now!” or “we will change your passwords for you” and stuck with that. Make up your mind, DH.
I changed mine, just in case. Also, I didn’t get any e-mail or notice about it, I saw about it on Twitter a few minutes ago in a tweet by @wordpress. If it wasn’t for that, I wouldn’t know.

I have changed my passwords for most of my important FTP accounts and I still can’t access through Cyberduck or Fetch.

Is there something else that I have to do?

Thanks,
Diane

You guys are awesome… If i were running dreamhost.com – six months would have passed and then maybe id have looked at the logs and said, hmm that looks weird maybe and do nothing about it.

EXACTLY the reason why im sitting in a dark room drinking bymyself and you guys are driving ferraris! You cyberpimp server ninjas

Can you comment on the fact that it appears you store passwords in plain text?

I just changed my passwords and on the next page my password was displayed exactly as I had entered it.

Wow… For the last 2 months my WP blogs have been constantly compromised, and DH blamed everything on me, saying I didnt keep the software up to date… its surprising to learn it was so widespread!

For this I pay $10 a month?!

@RG: i’ve been dealing with that forever. I couldn’t quite understand why it was happening and still can’t. Sucks how they make you feel like the idiot bc your not keeping your shit up to date…

When they say that the password reset will take *a while*, exactly how long? I’ve been waiting about an hour now.

How long does it take for passwords to propagate on average now? I’ve been waiting at least 10 minutes now…

I too, have had MAJOR Security issues. All from malicous code injected into EVERY PHP file, up to .htaccess hackings. Here’s the kicker – I didn’t have WP installed on my server. So no, it’s not only limited to WP security issues, it’s a related to a few terrible users on the shared environments, or the ones that simply don’t understand they are infected, that causes a widespread issue.

I’m moving all of my clients and projects to http://www.hostedcubes.com, this is the final straw.

I think DH had tried their best to protect our accounts. I will suggest encouraging these guys rather than giving them pressure. As this is something nobody want to see (except hackers), we should be more supportive. I don’t think any of us wanted to be treated by our clients in a similar way.

Thank you for your hard work! I believe you guys will fix this!

awesome, changed my pword now i cant even http://FTP... rad

OMG! LOL!!!! This is bad! It’s NOT possible! Dreamhost is dying!

i cant access to my ftp account…. hellpp!!!!

Just imagine the kind of trouble you would have all been in MONTHS or YEARS ago had I BEEN THE ONE in charge of security… Hahahaha…

Kidding, no single man would instill THAT much trust in me, let alone MILLIONS who have been provided (really incredible) service, support and hosting from these guys.

Good job DH for spotting the breach and documenting it.

After 6 years of DH… now i will sell all domains :(

Ditto to what Gabe said – why are you storing passwords in plaintext in a database anyway?

It’s not for convenience: after the confirmation page, the Control Panel doesn’t offer a way to view old passwords. (well – except for DB connections)

But in the trade-off between convenience and security, I’d much rather have my passwords stored with a company who old saved their hashes, and not the original. DH gets so much right, but then basics like this are still lurking around…

OK, what is going on?

I changed the FTP password, then after waiting about 10-15 mins for the pending icon to disappear, I STILL CAN’T CONNECT!

Well, this is even more of a puzzle. After changing the FTP password – and it showing as being successfully changed. I was able to log into WS_FTP no problem. So why is that a puzzle you ask? Well it logged in automatically with the old password – and it worked. So changing the password in the panel doesn’t seem to be doing much of anything if the old password is working.

Dreamhost is vulnerable and is not good. Dreamhost is dying. All planet will be affected… ApocalypseNow! I will shoot myself and i will sell all of my domains in next few days! :(

I demand a ferrari. i want a ferrari because i can’t log into my account through ftp at this very moment!

Changed the password as instructed. Now I can’t access my site via FTP using either the old or new password. I also can’t access my DreamHost panel. It’s been about 20 minutes. I know pwd reset takes a little time, especially now, but I think I fell through the cracks.

Without panel access, how do I report that I have no access? I guess I could go through the “forgot pwd” procedure. Probably best to wait a day or so for things to die down.

I have to say this was a ridiculous overreaction, and incredibly inconvenient to the userbase. The could have temporarily suspended access to FTP for all users, then investigated, then made a rational decision. Instead, they did this.

2012 = ApocalypseNow! = Internet LAWS! = SOPA = All planet will be affected! You want more?

Like everyone else I have been experiencing the htaccess problem and insertion of malicious code.
I want to add that I have had this happen on a url that stores nothing my .mp3 files. how can that be software related?
I feel for DH with regards to the hacking situation, the net has been insane the past 2 days with the SOPA stuff, then the MegaUpload and Anonymous stuff, now this!

So far the “abundance of caution” is worse than the original issue… I’m just going to wait a few hours… maybe until tomorrow… see if it eventually gets fixed. Been with ‘em over 5 years, and this is the worst I’ve seen.

Hacking situation on Dreamhost .. is unbelievable!!!!!!!!!!!!!!! For three days I lost trust with over 80% in DH, WTH? :|

I have yet to receive an email. Had it not been for lifehacker I would have never known. What gives?

Nostradamus predicted the end of the world comes in 2012 !! :|

@James @eric @gabe @mike @david

I could be mistaken, but I don’t think that there is any evidence that FTP/SFTP or shell users passwords are stored in plain text. Just because they display it on the next page doesn’t mean that it has been stored in plain text. It could just be an echo of the var you passed them before they hash it and store it.

I couldn’t find anywhere on the panel where you can retrieve passwords; I haven’t looked everywhere, but it wasn’t in the obvious places.

I sure hope they don’t store credentials in plain text! :)

Personally, I think they did the right thing. What, with the value of an ounce of prevention and all…

err, great. I just checked my joomla install and it’s gone tits up.

Dokuwiki on another domain is doing better though :)

Will you all quit your bitching? They’re working on it. The outcome could have been a hell of a lot worse and they’re being honest about it. You don’t see this out of most hosting companies. I’m just going to sit back, relax, have dinner with my family and maybe a glass of wine while I wait for this to be done so I can create new passwords later.

That said, it would be interesting to hear what exactly happened. =)

For those inquiring as to whether they should change ALL their passwords, here’s a message I saw when logging into my control panel:

“Please change any passwords ( Panel, Shell/FTP, Email, Mysql, etc )you may have with us.”

Are you guys going to change the cpanel password reset page too? When I click Forgot Password, it’s sent back in plain text. That can’t mean it’s hashed… I don’t feel secure and I’m very close to not renewing.

I’m glad they are taking steps to do something about whatever this compromise is. I think we should give them a chance to deal with it, and explain what has failed, and how they are going to deal with security in the future.

Myself, I had a bunch of sites in my account continually hit with php and .htaccess injections, I assumed from older wordpress installations that didn’t get updated in a timely matter. My fault. But after cleaning up it kept happening, and I’m reading about others here that also have had malicious code inserted into their accounts domains, even ones where wordpress was not running. It was always a suspicion of mine that the bots/hackers were getting in another way — and this announcement seems to confirm this.

I really do trust dreamhost as a paying customer for almost 13 years, and have seen them come clean about failures. I want to hear exactly what has happened here and how they are going to make sure security failures like this aren’t going to happen again, and I’ll be glad to be on for 13 more.

@Chris, LOL … i have only one server at my home and is more stable :))) No flood or hacking attempt :)) Think please… :|

K, just found the mysql db user. They probably shouldn’t store that in a reversible encryption method. The FTP/SFTP and shell accounts may be stored differently.

Only DH knows.

Any comments DreamHost?

Noone knows, all comments are stored in Dreamhost Database, but where is the database :) ?

Changed my password in the panel, still am unable to login after 30mins. Do you have an estimate of how long until i will be able to login with my new password?

Just adding my name to the list of people affected by the htaccess file. I removed all WordPress installs but still had common.php and coockies.txt being created almost immediately after I removed them and edited the htaccess file. Glad to hear others are affected so it’s not just me, but hope Dreamhost can sort this ASAP.

@leigh, ofcourse, I know, ask people from SOPA! :\

MTV is down too, check your TV’s asap…

…And the password changes are taking ages to take effect. Normally it takes 5 minutes tops. Been waiting for around a half hour. I have things to do. I am not pleased.

has anyone else had weird spam emails sent from their server name at some point?

EVERYTHING IS AUTOMATED ON DREAMHOST, no people’s there in that Datacenter … NO humans ONLY machines ! This page is for summary!

Just wondering, how long is the current ETA for a ftp password change?
Also, how do I go about finding if my files have been hacked? At what date/time?
Is there a site or a script that will examine them, similar to an antivirus? I use joomla and wordpress,
Thanks :)

i’ve been able to FTP the past 20min. now, so i would keep trying until you get in, or make some food, coffee, do drugs until some more time passes…

hey sweet, ten minutes after my new pw is working!

@GJ, LOL Joomla and WordPress are the most vulnerable web platforms :P Try Media Wiki!

Also stuck in password limbo. 30 mins+

ak47sigh er, yeah okay :s

@ak47, i have a few sites in Joomla and WP, what makes these vulnerable? and what are out there better than wiki, wp, J!? because as i see it, these are the top dogs of the game… and technically everything has some sort of vulnerability to it.

If you want a secure webplatform use in-house web platforms for websites and apps. too (I mean with your code) ;) I use PhP for this :D

Joomla and WP code .. everyone knows them … this mean = vulnerable code (php/ajax/js) .. etc.

agreed with shawn, the very nature and function on my site relies on joomla addons which I doubt will work on any other CMS since they are joomla addons. I’m not claiming to be the worlds authority on website development, but I’m pretty sure that reconstructing my site in another system would take many many man hours :s

Found this added to the top of .htaccess:

RewriteEngine On
RewriteOptions inherit
RewriteCond %{HTTP_REFERER} .*(msn|live|altavista|excite|ask|aol|google|mail|bing|yahoo).*$ [NC]
RewriteRule .* http://sweepstakesandcontestsinfo.com/nl-in.php?nnn=555 [R,L]

@Lee Yes hashed passwords can be sent in plain text. Done on all my sites but one and that one resets your password.

——
Everyone has been talking about having problems with code being injected into .php and .htaccess files and i have yet to have any problems and i have sites running WordPress and other popular scripts. Just lucky maybe…

I am in no way saying it is your fault but i would double check all plugins running on wordpress sites. Even if wordpress said they were safe does not mean they are i trust my own knowledge only. Yes hard to do if you know nothing about scripting.

As for custom scripted site double check your script.

Also scripts can be found that check for malicious code on all files under your account..

It is more better if you are not use any passwords for your websites … leave blank or put 111.

@A I would remove it if you can.

It happened on November 10th.

(Just found out about it now and November 10th was the last modification date)

It’s been more than 2 and half hours now since I reset my passwords and I still have the little pending icon and can’t log in with FTP / shell, what gives!?

It also wouldn’t let me overwrite the .htaccess file, I had to delete and upload a new one.

Everything is limited on shared hosting, like 90 MB RAM! You cannot instal IP boards :( On Dreamhost for IP boards you need a VPS :P

Just reset my password again, and it changed pretty quickly this time and I can log into the ftp. Not seeing any changed files at present :)

Mine also went quick when I tried again, but only shell reset, not FTP

@ak, thx ill keep the in-house stuff in mind…

It doesn’t appear that my sites’ passwords have been changed as of yet. So my question is, if I manually change them, will those changed passwords then get reset when they finally get around to me? I’m not really sure what I ought to do, and I’d hate to start anything only to get cut off in the middle.

The answer is, yes. I updated all of my passwords earlier tonight and they have all been reset by DH again…

@Lara I’d go ahead and change them anyhow. Better safe than sorry in the long run.

Hey, maybe if you guys at DH understood the concept of PROactive security instead of REactive security, this wouldn’t have happened. My web server is running on an OS with a kernel released NEARLY TWO YEARS AGO, PHP/MySQL several major security releases behind, and generally out-dated software. Seriously, do you people even care?

I’d just change the passwords again, it really did change quite quickly for me, less than 5 mins.. of course it might have been just a fluke

Did Dreamhost send out any email advisory about this issue? I did not get one. just found out via another random person on twitter.!

This explain a lot of bad things happening to all my websites!

Goddamit! I knew there has been a breach in Dreamhost security since the fall. I’ve randomly been experiencing problems with the .htaccess hack (pharma hack) on many of my sites here and report them. Dreamhost keeps blaming wordpress plugins and wordpress updates, BUT this only happens right after the auto dreamhost updates! I went in and disabled all auto updates and it appeared to stop.

So this is the 3rd time in two weeks I’ll be changing my FTP passwords. UGH. Hopefully Dreamhost can get this under control fast.

WordPress users, be sure to check your .htaccess files. They should look like this:

# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
# Prevents directory listing
IndexIgnore *
# END WordPress

If there’s anything else, then things are screwy!

To see your .htaccess file, go in to your FTP program and “show hidden files” up in the “VIEW” menu. Then duplicate the file renaming it with a .txt at the end so you can download it and edit it and reupload it. THEN simply delete the old .htaccess file and take rename the new file back to .htaccess ONLY DO THIS AFTER YOU HAVE CHANGED YOUR PASSWORDS (FTP, WORDPRESS AND DREAMHOST PANEL)

Hope this helps some of you. This is a really jacked up hack that I believe has embedded itself into the “auto” updating one click installs.

I cannot access any of the 45 domains via FTP on DH? This really is BAD.

Jack you need to change your ftp passwords before you can log in via ftp :)

I see now after going over this thread several issues with .htaccess – I did alert Dreamhost to my theory that the actual hack happened back in November 2011 and somehow got into the “auto” updating of their one click installs. This is why when you do a thorough cleaning, change passwords, suddenly when your dreamhost comes in to auto update it inserts the bad mojo. Pretty clever whoever did this.

Dreamhost should offer SOMETHING to those infected by this. Especially for all the lost time and saying it was OUR fault.

i changed my password, and my ftp / dreamweaver still don’t work,,, this is how i make a liveing

How long ago did you change it? Do you still have the timer showing on the user page?

I have the same problem as William – I have changed my passwords (and the clock icon has gone away), but I am still unable to log into my FTP accounts.

Retracted – my clock icons have not gone away. I thought they had but I must have been looking at a cached version of the page. Update scripts must be really hammered right now. :(

If its been more than an hour ago, try changing it again?

Which are the names of the servers affected? There was data leakage?

Changed my password, just waiting on the system to update so I can update my page. Only found out about this issue when I uploaded the latest version of my page today and got an inccorect password error. Have not gotten any messages from DH via email yet.

For those of you having to work through this debacle, you should wait till you can’t access the sFTP/SSH account. I have 11 customer accounts with well over 200 sFTP/SSH users that will need to be “fixed”. I was able to login to my personal sites for most of the day. Around 6:30PST sFTP with my previous password quit working. I then reset the passwords for those account and all is well. Those 11 personal sFTP/Shell accounts took me over an hour to reset via the manual process in the cPanel and test. Gotta love it.

I was hoping to be able to automate this via the DH Panel API and shell script but no user_update method unfortunately, might be handy in this case…

Jw

New passwords can go into effect. Clock icons in user panel never disappear.
+1 @Mutiny32 The software on Dreamhost servers are seriously outdated…

update, passwords are working now. It just takes longer than usual to change password in panel.

I really didn’t want to sysadmin another machine, but looks like that’s the only way to get it right. Still moving to Amazon EC2…

What the hell is wrong with you people? We are completely off line again.

You don’t support SOPA by any chance….Eeeek!!

It is awesome that Dreamhost didn’t bother to email me a single time to make me aware of the breach and FTP account reset until I randomly stumbled upon the issue a few minutes ago when trying to connect via FTP. Customer service, anyone? Between this, email downtime, server downtime, etc. I will be evaluating other options for all of my websites and domains upon expiration.

As much as I love DH, I cannot believe that I have to spend the night manually changing 50+ passwords now and then informing clients of the change. Sigh.

First of all.

Find it funny that every time something happens, I see the same chant by people (I will be evaluating other options, etc).

I just need to know something. I changed my password but it still says invalid. So is it taking longer than usual to reflect?

Yes I know, people, it’s inconvenient to change a password, but it’s also good security. On top of that, instead of complaining, I’m going to say THANK YOU to Dreamhost for being so quick to handle a possible security problem before even worse things happened to the users. Would you rather all the passwords be left alone and then a month later, those of you with web stores end up having your customer credit card #’s stolen? Come on, yes, it’s unfortunate, but this is the world we live in. Stop whining and be thankful we all use a host that actually pays attention to security!

I went in and changed passwords for all the domains I administer. Waited some time for the hour glasses to go away, and then try logging in. Worked on one the domains for sftp and another I was able to SSH into with the old password still. Now, neither the old password or the new one I set work, so is there some other process still happening, are they just hitting accounts to force the change or what? Should we wait for an official email to change them, or what?

I’m noticing downtime on the canopus server. Is that related to this issue or something separate?

Took about 20 minutes for my new passwords to take, but they do work now.

slowly .. but . surely .. moving …. away ….. from … .. dreamhost …..

Why have I not received any communication about this? I’ve been struggling to access my account for the last couple of hours and finally decided to go login to the panel where I was notified of the situation?

I’ve now changed my password via the control panel but I still can’t login.

This is very frustrating because I desperately need access to my account!

WTF…I had all ssh/sftp passwords changed and woorking a few hours ago and now brken again???

I’m really unhappy that I found out about this on LifeHacker, not through any communication from Dreamhost.

Thank you for taking the care and consideration for my privacy as I expected.

I run a clean WP install (NOT through the 1-click installer), but a fresh copy downloaded from wordpress directly and manually installed. No issues here. I’m guessing the DH version is infected.

I can now ftp in and all of my websites are gone.

And yes, it’s shitty to have learned of this from techcrunch, rather than DH themselves.

This is so messed up. No notification. FTP passwords have been stolen before here. I’ve had enough, but the lack of customer notification is the last straw.

My customers notified me, not my webhost. Not even an email.

I just want to say thank you to the Dreamhost team for being super careful and considerate of their customers security. Yes, it was inconvenient to change passwords, but considering the alternative I’m more than appreciative of the hard work put in by the Dreamhost team.

Shame the community decides to leave selfish, spoiled, obnoxious, and altogether unhelpful comments here. It’s just embarrassing.

Also, for all the people complaining they didn’t get any notification about this: It’s in a big, bright red alert-like box on your panel. How could you miss it?

Our WP pages were hacked last April. After much back and forth with DH they insisted it was our fault due to compromised FTP accounts. In the past few months our site (with WP removed) has continued to be hacked. At this point we’re planning on going to a new host and abandoning all the infected files on DH.

It’s disappointing for us being customers for many years. We remember the credit card customer hack of our billing information a few years ago and stuck by, but service has gone way downhill to the point of the responses to inquiries being hostile at times.

WTG guys, thx for getting it fixed. Keep up the good work. don’t stress the negative thoughts…

You should hash passwords.

Thanks for addressing the problem. Sometimes you just have to do the stuff that nobody is going to be happy about.
Thanks DH

And I found out about this via Twitter (Lifehacker) and not directly from DH??????….!!!! Odd don’t you think?

What follows is a summary of what *appears* to have happened according to the comments above. I may me wrong on some of the specifics, but then, it’s for DH to investigate:

I took the time to read through all the comments. While the dreamhost notice
was double plus unclear as to what exactly was compromised and how, the
comments gave a very good clue.

For the last few months, many many users had been reporting security breaches
on their web sites, only to be told by DH that the clients were responsible
for the security of their own software (scripts) by keeping them up to date.

Almost all of them complained about .php files and especially the .htaccess
files from being compromised and edited. The way the .htaccess files were edited
was particularly ingenious: it was made so that the webmaster and a regular
user would not be made aware of anything being amiss. However, when dealing
with a search engine indexing bot (google, yahoo, etc.), the .htaccess would
redirect any page on the whole client’s site to a site with malware, selling
pharmaceuticals, etc.. Thus Google, etc., would be, and had been, indexing the
spammers’ web site instead of the clients’, with the page rank of the clients’
web sites benefiting the spammers.
Several clients complained that their sites had been flagged by google for
dealing malware.

How did that happen? Apparently, the breach occured in November via the
one-click install wizard offered by Dreamhost: One click and your whole
Wordpress / Drupal web site is installed, ready to use, automatically updated
by the wizard. Apparently, it’s the wizard itself that was compromised and
anybody who used it was affected.

So, it appears that DH was wrong (above) to deflect the responsibility onto the
clients. It was the clients who were the most trusting in the tools provided
by DH that were affected.

However, I don’t blame them. In the hostile internet environment we live in,
shit like this happens! :(

Fortunately for us, our sites had been installed manually using the code
directly from the source, drupal.org.

I rapidly checked the .htaccess of our sites to make sure and everything
seemed normal.

Dreamhost, I like your company, I appreciate you stamping out security issues, but I’d like you to tighten up, improve your service and reliability.

January 20th, 2012 at 9:47 pmSamson L Jacson Says:

50 char passwords ftw!

It’s about time that you start hashing the passwords. Every time I hit the ‘Forgot password’ link, you send me back my password in plain text. That’s just silly.

Well, I have yet to receive any kind of F’n email from dreamhost telling me “hey, some jerk off is attempting and more than likely successful at hacking our database server.” Yet another reason for me to be disappointed with dreamhost.

I am unable to log into the webFTP app to reset my passwords.

I never received notification of this problem. I found out while troubleshooting my sensors and trying to figure out why they could not contact my FTP server.

Hey guys you should really encrypt all passwords…

they ARE sending out emails guys — thats how I got here! theyre probably all in queue, you cant send 50,000 emails out at once or email ISP will block your domain for spam. chill the fuck out.

My Website has been deleted, This is wired.

Would love to change the password on my account. HOW, please do I do this. No amount of searching has gotten me to my account to do this.

well, then, please/…how do I contact you folks???????????????

If my Dreamhost ssh/ftp password has been compromised and I use the same password on say my gmail account, is there a way that they can get access there if my web panel login username is the same for dream host and gmail?

I can not log in to the FTP for the life of me even reset my password a few times anyone else having this problem?

this is total bs that i learned about this by loging into my dreamhost panel, you should of sent mail out. I want a complete list of all the information that was stolen was our credit card numbers taken? was our email addesss exposed. you have not provided enough information.

I changed my password, and stil can’t access to my ftp.
And i never received an email from Dreamhost for telling me about this issue, i only saw the message in the panel…

Iann, same with me. I only found out about this issue when logging into my webpanel. I never received any email. I changed my password as directed but still can’t access sftp.

I told you about those issues with lots of shells several months now. Many of the shells were combinations of files from different accounts and that means the shells had root access.

Do person using dreamhost apps only, are also affected

So where are we with all these things?, I just received a mail about this MAJOR issue.

Do we still need to change each password? DBA password included?

Are all password need to be changed including of the databases?

As they did to violate your system? You do not have security measures?

this dialogue is bogus. i am the owner. please lock knock down the intruders. they clearly don’t speak american english. i’ve been the domain owner for 10+ years.

Here is what to do from investigation:

1. Block the domain ed-drive.net (72.44.82.14) – it is where all those pharm spam comes from. I think it would be better to prohibit any activities from domains registered at internet.bs pointed from topdns.com.

2. Block the subnet of futurehosting.com as that is abuse proof hosting for doorwayers and pharm spammers.

3. Block all domains coming from dnsmadeeasy.com as the hackers are using those DNSs

Time to move to somewhere that can protect passwords… cleartext? what *ARE* you thinking?!?!?!

Why does DH store these on file? We dont even do that in our applications. The only recommended way to store files it to hash it. And the only way to recover these passwords in plain text would be through the private key. And the only way a private key can be accessed is from a valid computer (IP restrcitions + smart cards). Perhaps DH will do that in the future?

As an example once I create a password, I need to keep it safely. And I can’t login to DH Panel to see it as a DreamHoster. If I forget I need to reset. And that is the way it should be. I am worried.

I’ll do a kindness and respond to the last 2 hours or so of comments since I figure the DH team is either asleep (except a night owl support ticket guys) or working on the issue.

@Brijesh K: Contact support via the Panel.

@crm: For the SSH/FTP users, as being required here, on the left side open the Users section. Go into Manage Users. From there, select Edit on the right side of the username you wish to change the password of. At the bottom, enter the new password twice and Save Changes. For your main account (not required here), go under the Billing section and Security. In there, enter the new password twice and press Change. If you still wish to contact support, go under the Support section and click Contact Support.

@Antonis: I don’t exactly understand the question. Are you saying your Web Panel password is the same as Gmail, or that your Gmail is the same as your SSH/FTP? Or is it that all 3 are the same? If its the former or the latter, Change Gmail and your Panel passwords ASAP. Refer to my above response for how to change your Panel login. Your passwords are hashed in the database but it is always better safe then sorry. For future reference, your main email account should never be another password anywhere else. It’s your last line of defense for regaining access to hacked/lost accounts and generally contains the most personal information. Your bank/credit card sites should also never match any other passwords (even one another if possible) as they contain your records among other things hackers would love to have. I personally have 1 password for each of my bank and credit card accounts, one for Gmail and then one for here. I use a default password on my forum accounts as I consider them to be throw aways if anything goes awry.

@Justin: Yes. It seems to be taking longer then it should. I have a feeling many people are just finding out about this and they’re being bogged down a bit.

@tom: If you had read any of the previous comments, ALL of your issues we’re answered previously. So to answer your questions… They did. Good for you. No. Probably. They’re still looking into it. There, all answered.

@Iann: I believe the servers are running a bit slow with all the people just finding out about this. It should be in effect by the morning at the latest. They did indeed send out emails, but the limitations on sending emails are so many per hour otherwise they’ll have issues with email providers marking them for spam.

@Fabien, Chaitanya: I haven’t seen any information regarding DB passwords being compromised. I doubt this is an issue.

@mascali: Just because someone violates your car, does that mean you do not have an alarm? Having security in place isn’t a guarantee that no one will get in. It simply takes care of all KNOWN methods of entry. There isn’t much you can do about unknown methods until they, become known.

@paul: You must not have read the previous comments. NOT CLEAR TEXT… HASHED. Common sense says that even HASHED passwords can be broken. Look up MD5 rainbow tables and you can understand that the so called security most web apps STILL use today is severely compromised. They aren’t using md5 by any means. BUT, as a PRECAUTION, they want all passwords changed. It makes perfect security sense

@Ravi: Again, read the comments. It is clearly stated by Brian at 11:08AM that they are HASHED.

Furthermore:

4. Block IP 184.154.34.92 and probably all the network of Inferno Solutions as all the injections goes from there

Do we also need to change all of our db passwords?

gina – I suppose yes as well as go check the database content as it could be changed be the hackers.

@Brian et al.

> Customer passwords are not stored in plain txt, they’re hashed.

And by “hashed” I assume you mean “bcrypt,” correct?
Normal digest algorithms, salted or not, are not safe for password storage.

If FTP passwords were compromised…

…. then the vast majority of people likely need to change their DB passwords, since they are probably using software that has them storing that password in plain text, in files easily compromised if their FTP was accessed.

And DreamHost really should be telling people to do that IMO.

I’d like to know why I haven’t received an email and needed to find out via Lifehacker. I’ve had an account with Dreamhost for over 10 years, and I assume that I will be properly notified when there’s a major failure of this kind.

Anyone else not yet received an email, despite the “resolved, all emails sent” tag on this issue?

Agree.. I found out by lifehacker also- what a crock..
I feel vindicated as several of my sites have gone down over the past few months and I get the blame- “just update your wordpress core files”.. do that.. BOOM go down again… rinse repeat.. I had all the pharma crap, and I sat manually going through all the php files and the sqldb but couldnt find where the injection code was… as for help as sites are not loading.. but dhost just pass the buck every turn to user error/ not updating etc..

Im glad im not alone with my dreams being pssed all over.

DH, thanks for info.
To all: do you know any tools to make security checks in linux (like rootkit detectors etc)?

Outstanding. Yet another issue.

I have changed my passwords and now cannot log in either via WebFTP, Filezilla, Nautilus or DW.

I didn’t get an email either. I’ve changed my passwords but they are taking a very long time to take effect (nearly an hour so far since I did the first one). Control panel shows a ‘clock’ symbol next the user names…

@Gingerdan
Thanks! Wish you a luck with your issue.

Aqui tem um passo a passo de como limpar o seu site WordPress se ele for comprometido.

Here there is a step by step on how to clean up you WordPress installation after he get compromised. (in Portuguese)

http://www.dowordpress.com.br/seguranca-wordpress-solucao/

Got notified by DH via email a hour or two ago. The password change took a while to take effect, but all seems fine now, so no complaints from my end. I’ve been a customer for 6 years, so far so good. Of course – every incident hurts, even with an IR plan at hand, but the important bit is the “lessons learned” afterwards. I’m sure DH will take note!

Hello,

There is a problem with all my passwords. The Shell, FTP and Panel passwords dont work and i used the recorvery of panel by my e-mail and did not work too. I need your help!

Show of hands – how long is it taking for your new passwords to update once you change them? Changed one of mine over an hour ago and still hasn’t taken effect…

[Update: looks like it is taking slightly over an hour for new passwords set to take effect]

Even if ftp/shell passwords are hashed the database passwords are not, not that it actually matters if someone has ftp access to the database info in your PHP scripts. Besides, the unhashed ftp password is still sent back to the user to display “Your new password is: blah” which is just another point of risk.

I’ve been waiting well over an hour. Contacted support but no response as yet.

This is the phenomenon we want to see

I changed shell and FTP usernames via the Panel. It said a few mins to take effect. Tried using the accounts half hour later and there is authentication faliure, when using the new password, or the old password. Seems the change password part of the panel isn’t working? I’m now locked out of FTP and shell.

Andy: in my experience it is currently taking a little over an hour for new passwords to take effect. I know it sucks, but give it another half an hour.

I advise everyone not only to change all DreamHost passwords (FTP/shell, web panel, mail accounts, database) but *also* to change the password of their e-mail account associated with their DreamHost account.
Today when I logged into my GMX mail account it told me there were two failed login attempts. I’ve never had that in years. Of course it may be a coincidence, but given the circumstances it’s better to change that password, too.

I believe that all DH customers DESERVE a compensation after this fiasco. How about it?

My Password is 123456, just like the Nokia. I change it into 123456, just like the Nokia.

i try to reset my password which allow SFTP/FTP plus ssh access, but 15mins pass away, it is still in pending status. And i can not login the server with ssh. Plz help to resolve ASAP. Thanks!

I clicked around to change password of all the users and now I feel dizzy :(

@Jimmy: The opposite problem also exists, and IMHO it is much worse: I have an SSH session which is open prior to the global password reset. It is still open! If the person who accessed the DH servers has managed to open SSH sessions, xe can still use them. This is a fucin’ MAJOR security hole.
I’ve contacted DH, and I hope they will fix this soon.

30 Minutes passed and still in pending status. How many minutes is “few” minutes?

I’ve been having horrible hacking problems over the past year on my WordPress site, specifically the pharma-cloaking hack that takes over my URLs, page titles and drops pharma-crap keywords everywhere. This seems to explain it all.

DreamHost barely helped me at all throughout multiple past hacks. The support was so bad that I signed up for VaultPress as insurance last summer. In fact, it was VaultPress that alerted me to this latest DreamHost hack, and they removed malicious files before even emailing me. Because of VaultPress’s alert, I logged into DreamHost, which is when I discovered the alert. I’m STILL waiting for the email from DreamHost on the hack.

WTF is going on?

Are there better hosting alternatives?

I only learned of this by reading Lifehacker this morning. No email notice from DH. And I had the same .htaccess sweepstakes redirect hack that so many others have mentioned (google blocked access to my sites). This certainly wasn’t caused by any out of date WordPress installations. Mine were all up to date. Fixing the problem took me a day and a half.

BTW, I then used the security tips here: http://www.mastermindblogger.com/2011/14-ways-to-prevent-your-wordpress-blog-from-being-hacked/ I hope these work.

I’m very much a novice when it comes to code literacy, so if there are other security tips you can share (with links to step-by-step guides), I appreciate it. Thx.

How do you find out if you’ve been hacked when you can’t access your ftp? I’m still waiting for the password change to take effect.

ALSO! As someone advised earlier, change the password to whatever email account is associated with DH.

I suffered a nasty “pharm redirect” with one client site hosted here. Got minimal help sprinkled with an attitude. Moved to another host.

It’s that simple.

How do you change the panel password? What about my gmail password? Does it need to be change too?

Well, I have downloaded all my data now… I think I will be moving to a safer place…

And btw I still haven’t got any info about this via email.. I found this out via lifehacker…
I don’t think all your customers log in to your panel all the time… If things like this happens, send out email to all your customers ASAP…..

I changed the s/ftp password. but after cannot access. I have to make access for update. what shall i do?

DreamHost – please consider moderating the comments, it is confusing to read.

If your site has been full of spam, or your blog hacked, it is most likely unrelated to this incident and your own fault.

How long are these password changes taking to go into effect? Been over an hour since I attempted to reset all of mine. Now, nothing works. I can login to the panel with a new password, but ftp, mail, etc. is all down for me here.

Can’t complain too much….I pay $10/month hosting and for that – you get what you pay for. Mickey Mouse security I am sure comes standard with a cheap hosting account on any host.

Great another great day for DH and its users. when start of day is by hearing bad news. weldone DH once again on giving away our all passwords to hackers.

More on possible ROOT CAUSE of malware injections into .htaccess files:

On 2011-09-22, I did an autoinstall of the Piwik stats application. Shortly thereafter, Google started marking my sites at DreamHost as containing malware.

I notified DreamHost, who said it was no doubt something one of my PHP and/or Javascript scripts caused, and gave me a list of things I could do to solve the problem (most of which dealt with updating to latest release, which was useless as I was at latest release on all of my stuff). I eventually cleaned everything up on my own.

What is interesting is that I do not normally use autoinstall but got lazy this one time and now wonder if this was the start of my troubles.

Is this coincidence or not?

I know there is no guarantee that DreamHost staff will view this, but maybe it will help someone else.

And chances are it’s NOT WordPress that is the culprit. If it is, why just DreamHost and not the 10’s of millions of other servers across the globe that have WordPress installed?

I changed password many hours ago and ftp doesn’t work yet!

I tried to change password again but without luck. My website needs assistance and I can’t do nothing!

I’m very dissapointed with Dreamhost, you shouldn’t write that the issue is resolved when it is still unfixed!

I think we should be refunded for that.

No access to FTP, all site files deleted, no support, no chat, cannot do a damn thing. Great

I have changed my ftp password ten minutes ago, it still not reflected, no indication of how long it is going to take.
TechSmartLife

Your site is showing like this. What do i have to?
i have to change FTP pass? or SFTP?
SFTP means Shell/FTP? If it does, why you are saying FTP Shell….

User Account Type:

FTP account – file transfer access only.
SFTP account – allows SFTP (SSH ftp)/FTP file transfer access.

> If things like this happens, send out email to all your customers ASAP.
They did. But they have to limit the amount of mails sent during a certain time frame to prevent DreamHost from being black-listed by other mail servers and ISPs. It certainly wouldn’t help if e-mails by DreamHost would end up in people’s spam folder.

> How do you change the panel password?
Log in to your panel and on the left menu click Billing, then Security.

> What about my gmail password? Does it need to be change too?
Is your GMail e-mail address associated with your DreamHost account?
Is one of your DreamHost passwords (FTP/shell, web panel, e-mail, database, etc.) identical to your GMail password?
If both answers to these questions is yes then change your GMail password, too.

> How long are these password changes taking to go into effect?
Several hours.
Changing my FTP password took a little bit less than an hour, but I’m still unable to access my DreamHost hosted e-mail account. I changed the password about three hours ago and at the moment both the old and the new password are being rejected by the mail server.

Here’s what I don’t get: If FTP/Shell passwords were changed, why are all of my sites offline? Every site has its own MySQL DB password, its own user with its own password, but everything is offline.

I’ve already sent emails out to clients for this upcoming week so I’ll have ample time to move all of my domains to another host.

One more point i would like to bring here. I upgraded to wordpress 3.3.1 (after the security fix) manually and my upgrade is finished. But couple of weeks later i received one more email saying my request to upgrade to 3.3.1 is done and the old site is backed up with .old suffix, though i didn’t made any such request (since i already upgraded manually). Does everybody got such upgrade email weeks after you upgraded manually?

TechSmartLife

This has been a very bad with for Dreamhost, but at least they have us inform. At the company I work their hosting is at Networl Solutions, expensive, and when they are out, and this happens more frecuently than Dreamhost, you don’t have any place to check what is happening, all you can do is to call them by phone!!!

I also changed my ftp passwords but I still can’t login:((

I see I am still hungry, in the first line it should say, “This has been a very bad week for Dreamhost”, sorry for the typo.

I also had my .htaccess file hacked about a month ago. I reported as much information as possible to DH but received no human response (I got an autogenerated response along with an automatic scan, but I had since found the offending javascript on my own). I had some out of date themes, but I wasn’t using any of them. I figured it was a WP security issue and stopped worrying after the latest update.

However, it now seems that people who weren’t even using WP also have been hacked, and that this is going on for months. This helps me to understand a particular incident a few months ago where I received a call about a church website that I run. The person said that they couldn’t access it through their corporate firewall because it had been marked as containing pornography. Apparently the earlier version of this hack only injected the links for crawlers and bots for the purpose of SEO. But in December it was more obvious as the links showed up for anyone.

So how long has this hack gone on? Have our passwords been compromised for months? Why didn’t anyone at DH connect the dots to see that a lot of people were having .htaccess hacks across a number of products?

A comment above has more info: http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/#comment-265213

Ok, so I went on my Dreamhost panel and changed my ftp password. Now I can’t access my ftp at all. I’ve waited 10 minutes for the system to update, guess I can wait a bit longer…

IMPORTANT!

Within your WordPress go to the tinymce directory (/wp-includes/js/tinymce/themes/advance/skins/) and check all /img/ directories there. If you find any .jpg or .png files over 100kb size (usually 140-150kb and 170-180 kb) it means you have a hacker’s shell there. Just delete those files.

there are three types. what do i have to do? anyhow i changed my ftp and sftp(is this shell/ftp? or ftp/shell which you indicated.) again and again… but still cannot access. I have to make an access to update our website… please respond quickly.

User Account Type:
FTP account – file transfer access only.

SFTP account – allows SFTP (SSH ftp)/FTP file transfer access.

Shell account – allows SFTP/FTP plus ssh access.

Shell Type:

Dear Dreamhost:

I received your email this morning and have been following the Dreamhost Status website. However both seem vague to what happened and how we should deal with it.

Here are some questions I would like answered:

1. Do I need to change my Database passwords? I have ALOT of websites hosted on various VPS’s and this will take ALOT of time. So I would like to know if it is necessary before undertaking the task.

2. The “alert message” under Support in the Panel says I should change every password for every service including databases and uses the term “etc.” like I am supposed to know what to change. This is an unacceptable security alert because it causes panic without clearly stating what to do. It is also not the same message that was sent out via email which states only Shell/FTP/Email passwords need to be changed.

3. Did hackers gain access to any of my Databases or just your Databases?

4. What exactly was stolen? Again Dreamhost has been very vague on this point – leaving us wondering if anything is safe. I need to know exactly what was stolen so I can alert my customers if necessary.

5. Just as it is your responsibility to inform US about what happened – I need to inform my customers who host with you either through their own account or on one of our VPS’s what exactly was stolen. This is impossible to do because there has not been a clear statement as to what the breach entailed and what was potentially stolen.

6. PHPMyAdmin – Is there any way to limit access to PHPMyAdmin to an IP address? eg. IP Address Whitelisting? To me this seems like one of the most unsecure aspects of Dreamhost hosting.

Thank-you for your time.

Two questions:

Does this include VPS ssh/ftp users too, or just shared accounts?

Did this affect any scheduled cron jobs on any of the accounts?

Thanks

I changed my password 2 hours ago and the system is now accepting the new password and I can see the FTP directory.
But, when I try to transfer a file to the directory I get an “access denied”(550)//
Anyone else having this problem??
Thanks

I have a similar problem to others: I changed my shell/FTP passwords on both sites that I manage and now cannot access FTP on other; I just keep getting ‘login incorrect’ error messages. I changed them yesterday and still couldn’t access them this morning, so I just changed them again and still no joy.

I want to take a second to thank DreamHost for disclosing the activity and taking action. I would much rather have to update my passwords than learn the hard way that my account had been compromised!

update, did the ftp reset thing again and I can login to my ftp now and I can upload and delete files fine.

Would have been nice to get an email from DH about this.

I updated my passwords over an hour ago and am still being denied FTP access.

What’s wrong with?I was see my site have been changed by hacker.I was delete some shell tools in my site.

I was kill all shell account now.And i was chang my password at last times.

The changed password took about an hour to be effective, but works now.

Does anyone know if there is a way in the ftp to view files by date rather than alphabetically? I just want to see if any were modified recently.

January 21st, 2012 at 6:18 amAuro Florentino Says:

Are you guys storing our login passwords as plain text on a database?? Really??????

I’ve updated my FTP password but still can’t get into to my sites! How long should this take?

I just closed my account at Dreamhost!

I’ve been trying to become a blogger for days now but the red tape is incredible! I just can’t understand why so much talk about becoming a blogger. It is confusing particularly for us who are not technical bugs on the ciber/computer thing. The othe day I applied for the “dreamhost thing” and I can’t even remember what it was about. Can you bring me up to date if at all possible? Thanks. Felix A. Rodriguez

I changed my password using the web panel and still can’t access my server ?!
Are you still working

You’re storing our shell passwords in your database? Unencrypted? Why?

How long does it take to update password? I’m vaiting for few hours now…

Is this a F’in joke! What a useless company, defo leaving you guys. I thought about it before but didnt want to got through the hassle but now you guys have just topped it off! Goodbye Dreamhost

If you have a Twitter account, recommend you start following #Dreamhost and #DHStatus. Word gets out a lot faster there than on email.

Why is there a password length limitation when changing passwords if the passwords are not stored in plaintext?

As mentioned earlier, the “Forgot Password” feature on the DreamHost panel sends the account’s password in plaintext through email. This means the password has to be stored in cleartext or some form of reversible encryption, which is not secure.

Either DreamHost is lying to us or they have decided to store an even more important password in plaintext. Time to come clean.

I had to wait about an hour for my new password to work. But after that I had no problems connecting with FTP.

I am also going to update other passwords as well (mail, web panel, mysql, …) to be certain that no one will gain access without me knowing.

I tried changing my shell password, but it’s not taking effect.

My *OLD* password still works! What does this mean?? Did you fail to ‘reset’ all the passwords? Your message above (and all the updates) say that you reset ALL passwords. Mine wasn’t changed – does that mean my server was NOT compromised?

I’m using VPS. When I login with ssh, hostname says “ps4696″.

January 21st, 2012 at 7:04 amabout_hashing... Says:

If those passwords were properly hashed, there’d be no reason to worry at all. I won’t do any more business with you.

When I changed my password I got a message stating “Your password has been change to” and then my password. I’d assume that means the password is being stored in plain text or using reversible encryption somewhere. Will the password system be updated to use non-reversible encryption at some point?

January 21st, 2012 at 7:05 amMichel Carvalho Says:

Even I updated my shell users passwords I can’t access my account by ftp.

I updated my password. But I STILL can’t log into the webFTP. Neither my old password nor my new password (which I changed twice) works. What gives? What is going on? I REALLY need access to my website. Is anyone else having this problem? How is it fixed and how long did it take?

Paul Bartholomew – I think DH only updated Shared users, not Private Server or Dedicated. It’s not clear if this is because there was no need, or they have overlooked PS & DS. I *hope* it’s the former.

Im also having problems, I changed my passwords over an hour ago and still cant get in with the new or old passwords. This is pretty frustrating!!

Jake: I know for a fact thaht user password updates were taking just over one hour to process a little while ago. Email passwords are being updated much quicker.

Did anyone else here take the precaution of updating email passwords at the same time? I did, and they *rolled back* about 10 mins ago to OLD passwords. Happen to anyone else?

Changed my password about half an hour ago, but the server won’t let me log in through FTP or shell?

January 21st, 2012 at 7:20 amTutorial on hashing Says:

As you clearly don’t know how to properly hash passwords, here’s some free advice.

1. use salt
2. use a decent algorithm
3. hash the hash again a few times, to make the whole operation take longer

This way, even simple passwords are very well protected, unless the password is “123456” or “password”. That being said, I totally expect you to make the same mistakes again, like 90% of technically incompetent companies on the web.

January 21st, 2012 at 7:20 amjoseph phillips Says:

I have updated my password several times, and I am still not able to access my website. When will this be fixed?

Trying to change my password but keep getting an error message saying I’m using illegal characters. Even if I use their “Pick a Password for me” option, I still get the error message.

So now I’m locked out of two websites – not good.

Where are you Dream(Nightmare)Host????

Haha so many ignorant impatient assholes in this thread… thanks for being so quick to lock down our accounts Dreamhost :D

I’m using nothing but letters to change my password and I get an error message saying I’m using illegal characters. This is bad!!!

I’m always amused when a little hiccup in service like this happens and everyone starts bitching about DH and threatening to jump ship. I’ve been with DH longer than I can remember and this is the pretty much the same reaction every time.

Please do jump ship. Go to Gatorhost or wherever else (where you will have the exact same problems)… that will free up some more CPU power for me. :-)

Jumping ship is an entirely appropriate response to having all of your shell passwords compromised because your host didn’t care enough or wasn’t savvy enough to hash the passwords before placing them in a centralized location.

Since I got your email to update FTP password, I did but still fail to access the ftp.

Help me?

umm

oy

cant access email at all to make pw change email is down.

i changed all passwords
but fail to access the ftp

Thank you for being conservative and taking these security measures. I appreciate that you take security seriously.

> I did, and they *rolled back* about 10 mins ago to OLD passwords. Happen to anyone else?
Nope. I changed the mail password more than 6 hours ago, but the mail server is rejecting both my old and new password now.

> I have updated my password several times, and I am still not able to access my website.
It’s impatient people like you who slow down the password changing process for everyone.

> Trying to change my password but keep getting an error message saying I’m using illegal characters.
Had the same issue. The web panel is pretty buggy when it’s coming to changing passwords. It also states that it is accepting only a very limited range of special characters, but in fact it *will* accept even characters used in French or German language. And it doesn’t even state that the password is limited to 30 characters. They really need to improve it.

Just an FYI –

I contacted Support directly (using the web form on cpanel). I told them that I wasn’t able to change my passwords, and that my ‘old’ passwords still worked.

They responded within a few minutes, saying that they ‘manually updated them’ for me.

I didn’t tell them my NEW passwords, but those are now working. So, they were in the system already (plaintext or otherwise) – just not ‘applied’ until I contacted them.

So, rather than/in addition to posting on this thread, I’d suggest contacting them directly. It worked for me.

January 21st, 2012 at 7:44 amParanoidCustomer Says:

Maybe you should not display my password in the panel to anyone looking over my shoulder after I’ve just changed it?

I’m wondering if the passwords were stored plaintext or encrypted? I need to gauge how much time I have to change all the passwords, since there’s literally hundreds I need to change on my account. If the passwords were stored encrypted, then I don’t have anything to worry about, I used some pretty strong passwords, it would require the hacker several years to crack just one of them.

I changed all my FTP passwords last night. I see the message about changing all the email account passwords as a precaution. Is this meant even for those with their accounts hosted at Gmail servers?

Thanks for the email. However, I am NOT able to login at panel.dreamhost.com. Clicking on “Forgot password?” doesn’t do anything either (just reloads the page). What do I do now? (I’m using Chrome on Mac.)

I changed my password about 12 hours ago but still can’t access FTP. Also, all my sites redirect to bee . pl when you click on them in google. yay!

I’m at 2 hours and still waiting for my shell account password change to go through. Makes me wish I installed my ssh keys when I had the chance. Semi-patiently waiting.

I’ve seen several condescending replies which infer that the passwords were not hashed. What makes you think that they were not? Has DH actually said that they were stored in cleartext?

Also, I see a reply above that states, “If those passwords were properly hashed, there’d be no reason to worry at all.” Who on earth believes that?

Well, I´ve changed my passwords and it started working 30min latter, when the little clock disapear from the users.

Seens to be fine so far.

Glad DH sent me an email about the situation, and also, glad DH didn´t try to hide what was happening, I know a few companies (in Brazil, where I live) who would NEVER assume something like that, they´ll just sit and stay as quiet as possible.

A quick question… Any actions were taken in order to avoid this issue to happen again?

Thanks!

Okay, I just got in using what I thought was an “old” password (must’ve changed a different one when I thought I changed that one). Resolved.

As a professional programmer who has been in the hotseat myself for issues beyond my control, I would like to point out that:

a) I am only paying 6 bucks per month to host a web site, which, for the past 1.5 years has had almost no down time.

b) To be fair to thouse unhappy campers out there, my website is not-commercial, and a hobby, not tied to my job. No money is lost if there is any down time, and I do sympathize with those of you whose jobs/accounts are on the line. Try going to a customer and saying “it’s not my fault, it’s the server…..”.

However, I reviewed the actual hours between the reported breach and the resolution, and I am impressed; it’s not as if people were sitting around for hours doing nothing. Also, I would like to point out that NO SYSTEM is 100% invulnerable, given the very nature of the Web and its total connectivity all over the globe. Frankly, I’m just glad that the situation was not worse. For those of you who wasted hours and hours trying to resolve problems, you have my complete sympathy, and I’m not necessarily saying that Dreamhost should be completely off the hook. I’m just saying that we ought to give the techs there the benefit of the doubt; my guess is that they sat their for 12 hours straight with management breathing down their necks.

Someone registered a domain and an account with my email address as I received all the details by email, what should I do?
I hope they didn’t use my credit card info!

resolved?
i think not

If U R own database/systems not secure how we can assume security of our DB :<

I changed all FTP password about a hour ago
but still unable to login with new passwords

oh… paid my yearly fee one day ago
&my friend told me to change host… he was right

“Posted (January 20th, 2012 at 9:25 am PST) by Oscar”

“Subject: Information on your DreamHost account – please change your passwords
From: do.not.reply@dreamhost.com
Date: Sat, 21 Jan 2012 6:23 AM
To: KenSaunders@************.com”

Will you please tell me why it has taken so long for me to receive this “IMPORTANT INFORMATION” that contains “IMPORTANT ACTION REQUIRED”?

I’m a little pissed about that.

All of you guys asking questions on a comment post is stupid, if you need help don’t bother asking it here.

thanks dreamhost for letting everyone know, again this is a good thing for everyone that we take precaution, and this doesn’t mean people’s passwords were compromised.

I’m with Jake Rutter on this.

Changed the passwords some time ago, they still show as pending and I cannot get into my FTP.

This was a disaster. I’d like some more information from DH on what exactly happened. They need to come out and just say what exactly happened (i.e. Yeah the FTP database was hacked).. Shit happens, they’re a great host, I’ve been with them since 2007 never had a problem, I just want some more information into what happened.

Anyone claiming they’re leaving, great, more space on the server for me.

Been with DH a long time. Have almost 100% of my sites hosted with them. This is the first time in my experience they have been hacked. My sites never go down, the pricing is great, the tech support is responsive and accurate. In all, my DH experience is superior to that which I have EVER experienced at ANY other web host or service provider. It’s unfortunate they were hacked. But now that hole is closed (hopefully). I would like to know more about how the hackers got into the system. Hopefully there will be a follow-up forensic post.

I don’t think this IS resolved. I followed all the directions, changed all my ftp and email passwords (and used copy and paste to make sure they were all entered the same way). FTP is fine, but now I can’t access any of my email accounts. I’ve tried pasting the password, typing it in manually, over and over, waiting a few hours in case it took awhile to propagate, but I just keep getting “The server rejected the password.”

I’m still locked out of FTP. When I use fileZilla, I get a error:
Error: Critical error
Error: Could not connect to server
When I use WebFtp off of my panel, nothing comes up in the AjaXplorer window.

@Ken Sauders,

They emailed ALL their customers. This is WHY it takes so long.

@joejoe
“All of you guys asking questions on a comment post is stupid”

Forgive me for being stupid and correcting your sentence.
“All of you guys asking questions on a comment post *are stupid”
I’m not exactly sure what a “comment post” is, so I can’t address that.
“thanks dreamhost for letting everyone know”

I found out a day later. Not acceptable when it’s a security issue.

The author of this post (Oscar) -has- replied to some comments, and, if they didn’t want to, or expect to hear from customers, comments wouldn’t be allowed.

I posed a legitimate question and I resent being called stupid.

I’m on liberty.dreamhost.com and changed my password to my ftp account and can’t get in. I also created a new ftp account and can’t get into that one either.

@Renato Frota
It takes a day to send email? Bull. Spammers, mailing lists, newsgroups, corporations, media outlets, all do mass mailings and it never takes a day.

@Ken Saunders
You’re the man!!!

@RBL
Fully agree on all accounts. I just don’t believe that it should take a day to notify a customer when it pertains to a security issue.

Could someone tell me how to disable password access to shell (SSH/SFTP)? I setup the public/private keys as shown here: http://wiki.dreamhost.com/SSH Password-less login is working fine.

To people with unrelated website hacks: you got infected originally through old software (or world-writable files). You get reinfected because they put in a php back-door file to re-access.

Delete all unknown php files in all folders
Change all file access to remove world-writable permission

None of that is DH fault (but maybe they could start offering a script to disinfect – DreamAntiVirus)

This is pretty bad.

I’ve had ll sorts of problems with Dreamhost in the past month or so — hacked websites (pharma, various php files being modified, etc). Like most, I was told it was related to out-of-date WordPress installations, but they were all up-to-date.

Also, the Dreamhost password generator page has some problems: it generates passwords that aren’t allowed by their system–i.e. it allows illegal characters, two consecutive numbers, etc. I had to generate passwords several times to get ones that were OK by their own standards. What a drag!

If you’re looking to switch your hosting (since DreamHost seems to be janky and not secure), go ahead and check out LivingDot. Great folks & pricing there. Use coupon code “BLOGGING” at checkout –> http://bit.ly/y9EQLX

Why on earth would you store the actual password? Isn’t it a no-brainer to store a hash of the password?

If you’re paranoid don’t forget to change your RSA keys if you used them to log in securely to your site. Assuming someone got access to the hashed passwords and had time to access your account your key could be compromised.

http://wiki.dreamhost.com/SSH

My new ftp account is working. I received an email on it but it took a few hours. My old account isn’t working – perhaps I need to wait for an email on that account too.

I forgot to add that if they got access to the hashed passwords and had enough time to crack them first. Which my guess is fairly doubtful.

>If you’re paranoid don’t forget to change your RSA keys if you used them to log in securely to your site. Assuming someone got >access to the hashed passwords and had time to access your account your key could be compromised.
>http://wiki.dreamhost.com/SSH

I just updated my FTP passwords and was concerned because I was unable to connect. But I just waited about 10 minutes or so and now it’s working fine. Doesn’t look like anything on my numerous sites got compromised. FYI, I previously hosted with a smaller hosting company that was MUCH more expensive…and my sites got hacked there — and they never even knew it until Google quarantined them for malware. I discovered the corrupted files and changed them. The host told me that my passwords must have gotten hacked and that I should use stronger ones next time. I did. At Dreamhost instead!!

Ignore my previous post on RSA keyless login. They would only have access to your public key not your private key.

Good morning,

As you suggested, I created a new FTP/shell access password, 15 min or so. As I tried to publish my website pages, through rapid weaver, I was asked for my new FTP/shell access password which I entered yet, the publishing failed. Do you have any suggestions?

Thank you

at 11.08 on 20/1/12 Brian posted:
>Customer passwords are not stored in plain txt, they’re hashed. Unauthorized access to hashed passwords is still serious, however, and we’re treating it as such.

So calm down folks. It will take a while to run all those hashes past the rainbow tables. Also since Dreamhost acted promptly with a global reset, the hashes are now already useless even if decoded.

@Tom If you look back through the replies, Brian says that the passwords were NOT stored as clear text–they ARE hashed. (2012-01-20 11:08)

Hashing is like a padlock: it doesn’t prevent someone from breaking in–it just buys you time… enough time to change your password before the intruder can crack the hash.

@A the htaccess mod you posted above has the same format as an attack I recall from a couple of years ago. Note that that attack also added obfuscated code to other pages with guessable names like index and contact, if I remember correctly.

The intrusion was by automated ftp access. I eventually concluded that in that case, the crims got the passwords from malware on my own dev machine which accessed them from filezilla – it’s a known exploit.

That was not at Dreamhost. The hosting company where that happened now provides the facility to lock down ftp and sftp access so it can’t be used unless you unlock it first from your CP. That’s a routine nuisance of course but enhances security substantively and I think it’s worth the pain. It may help if you disable ftp access and only use sftp, AFAIK those attacks use ftp not sftp. I think there’s a checkbox for disallowing ftp in dreamhost CP.

The password change job must be taking longer that usual… I’ve been waiting about 20 minutes, my new password still not in effect.

I reset the passwords hours ago and still have a clock next to them under manage users. What’s the deal?

I can’t login with my newly created FTP passwords…It usually takes just a few minutes for the password change to update and it’s been hours already..

Anyone who thinks another host would do a better job just needs to visit their status equivalent pages and read some of the comments there. Great post by Ian, NO host is 100% protected against attacks.

I changed the ftp passwords about 30 minutes ago and I am still getting an ‘incorrect login’ message. WebFTP leads to a blank page.

I changed my passwords almost 2 hours ago, and I still cannot access my files using Filezilla or https://webftp.dreamhost.com/. I was in the process of posting material for a class that starts on Monday: very frustrating…

I am not receiving any email after changing the password. Is my email also compromised ?

I have noticed (and been remiss in reporting) PHP includes of short base 64-encoded PHP files as well, which display ad junk when search engines view the page. This has been happening occasionally for months now. Custom site, no WordPress or anything else. The changes are in the file system, not the database. I have a feeling this was, perhaps, a large-scale and long-lived hack.

Noon on Saturday, January 21st. Changed password. Still can not publish changes to my website. Please advise.

I changed the pass according to the instruction and waitted more than 10 mins, I still can’t login to the ftp/shell my account. Waht’s wrong?

I could able to login with my new password. I did contact the support with the issue.

It took about two hours for my new passwords to take effect.

To all the folks threatening to leave DH, please do. I don’t want to have wade through any of your ill-informed and inane comments again.

To DH, thanks for being proactive.

I asked a top-priority question to DH Support on 16.Jan. about this (#4682595) but STILL didn’t get any response at all :-( Whatever happened to the great DH Support?! This is most disappointing.

I changed my password, but I’m still not able to login…

Does any one getting email after changing the password ?
Some one from DH support please clarify, why we are not receiving email notification. if the email id is compromised then there is no use in changing the password. I might be over conscious but anything is possible.

Password Changed HOURS ago, still no FTP login possible. Password did not propagate.
Nightmarehost!

Why the f#ck were they stored in plain text? This is UNACCEPTABLE for a hosting company!!!

I’m finding a new host.

I changed my password too. I also still cannot login.

The hacker that took down some of DreamHost web site is a known Israeli hacker name Ofer Sadot. He was born in 1956 and live in a city called Hod-Hasharon (in Israel).

His IP address is 192.116.227.130 .
He owns a web site for anonymous email sending and spamming (and some Porno and Pedophile, too):
http://www.noipmail.com

This hacker is using some proxies, but he can not hide for ever.
Some of the HTTP headers exposed him by his real home computer IP address (Cdn-Src-Ip, X-Forward-For, X-Real-IP, X-Client-Ip).

The Hacker Ofer Sadot ISP is “012Smile Telecomm”:
http://www.012.net.il

I used to call 012Smile and complain about this hacker, but they did nothing. The Israeli Police arressed him sometimes, but he is keeping to hack and spam web sites.

The past week he tried very hard, with some infected computers, to hack DreamHost servers.

Took almost 2 hours for my shell/ftp passwords to update. My backup user isn’t working though.

I also wonder if this is the result of the wonderful @anonymous hacks that have been hitting government and web hosts the last couple of days! They said it’s going to get real bad and affect many others besides us.

Wow, what a hassle. I wouldn’t be surprised to see Dreamhost lose 20% of their customers.

We should try to be more appreciative of the efforts put forth by DreamHost. Not just this time, but anytime there has been any problem – they are on it – they are updating us on it – what more can you ask for? Without a doubt, all of us have dealt with their support department at one time or another, and who here argues it isn’t first rate?

Wait – maybe you want to shell out millions and millions on server centers and employees and code.. oh wait – we all pay per month because we DONT have the capacity to fulfill those requirements, and even if we did (monetarily) – how would that solve your problem should you find yourself in a similar boat, without dreamhost paddling the oars? KAPUT!

This is quick, pro-active response-time to what may not have even boiled down to such an intensive security breach …. but it *could* have.. be thankful these people know what they’re doing.

Thank God for DreamHost. Smart People looking out for us. That’s all I can say.

And to top it off.. I doubt many of you actually fathem the true workload required to resolve or remedy or safeguard against these occurrences at this time. Clearly this isn’t a one-man job. This isn’t something you drag and drop to resolve. This isn’t as simple as a email address. When the power goes out at home – you buckle down and wait.. Be earthy, do something other than stare at your lifeless screen all day. Play with the kids, take out the garbage.

Or – as many of you say – go switch hosts.. or better yet.. start your own hosting company.. lol.

Everyone *exhale* and try to be mellow :)

Looks like it’s time to move.

Why did I not receive an email about this until 5:07 AM this morning… nearly 20 hours after the breach?

Thank you for being proactive and informing your customers about this potential breach. Sounds like most people want to find some giant web hosting company that likely would have swept this under the rug and wouldn’t have bothered to inform anyone. Would like to see a little more preparation and testing of the control panel under periods of high traffic though. Thanks guys.

Success logging in! Thanks, Dreamhost!

I still haven’t received my notification. Good thing I picked this up from another IT info email!

A hash is a one-way process that makes cracking unfeasible (excepting md5, which apparently is a weak hashing algorithm). There is no key to a hash and no decryption process. The cracker has to know the algorithm used to make the hash and the salt, and then has to run every possible guess through the algorithm until their hash matches the target.

Caveat: Weak passwords considerably lower the bar.

It is possible that crackers have seen DH hashing scripts or DB code for storing passwords, so all they need to do is run their brute force processes and hope for some weak (easy guess) passwords.

Hush all you bitchers and take your hosting elsewhere if you want! More space for me! Good luck finding a host that will never EVER be hacked!

Agreed that dreamhost was very transparent!! Most hosts would sweep it under the rug and just answer some support tickets to hide it.

Agreed the email could have been sooner, but it takes !TIME! to understand the problem and where you are hacked before you just do a universal password reset and email all customers!! Did you people see the support page????? It’s frickin play by play folks and sometimes updates within 20 minutes!!! Unbelievable support!! Now granted the password reset isn’t propagating very quickly at the moment, but give it some time! Everyone is doing it at once!

Impressed!

I found it took a couple of hours after updating my user password before the new password took effect. Be patient.

Well I have to say I was in a bit of a panic this morning, but everything is up and running now. Changes to passwords done and operational. Dreamhost email received. For those of you still in a tizzy, you probably weren’t involved in the PS3 security breach and down time debacle of 2011. I’ll take the less than 24 hour Dreamhost response & resolution over Sony any day of the week.

Good morning,

Publishing my website pages with the new FTP/shell access password is fine now.

Thank you the DH Team ! : )

So the question now is this. When will DH stop storing passwords in plaintext?

The first clue to whether a website stores your password in plaintext is to do a password recovery. Websites, which store hashes instead of passwords, change your password, send a temporary password and tell you to change it.

Dreamhost sends the text of your existing password, which means they store them in plaintext.

Why does DH need to keep these passwords in plaintext?

Has anyone who DOES NOT have WordPress installed on one of their sites had a security issue? or had anything modified? I see a trend in the posts, and was just wondering.

Does the following quote from another site make sense?

“It’s a bit weird. You see, Dreamhost stores your account password in a hashed form, and that’s set. Your cPanel/SSH/FTP password, however, is different. They allow you to pick whether or not you’d like it hashed (apparently some people prefer to get their passwords sent back in cleartext).

To quote a Dreamhost representative, “If having user service passwords stored in our database bothers you, log into a shell account and change the password with “passwd”. We’ll pick up the new password hash on our next user config and store the password internally as “changedbypasswd”.”

That’ll cover you.”

This is what happens to any organization whose CEO is a lawyer.

BTW, where is Simon Anderson these days? On vacation in Hawaii?

@Kenders

I think you are the tool here. again, this place is not for asking technical questions and any questions regarding this topic should be directed towards the support team not here.

please do go on, you are such a little attention whore, why not write a blog post about it instead of bitching about it here.

fuck you little piece of shit.

Changed all my passwords… seems like it’s taking forever for the systems to catch up. I still cannot connect via FTP on all my domains. What’s going on?

So… I’m not able to do nothing, I can’t log in on dreamhost panel or my only website this annoying, how the hell am I supposed to change my account stuff?

First of all ftp is horrible and should not be used (” I really wish it would go away “).
Second of all what is so hard about making it so your users can upload a public ssh key in the admin and skip making a password all together?

If you admin system is accommodating for the lowest common denominator (“noobs who dont know what they are doing”) vs. best standards and practices, then you have no one to blame but your selfs.

A good example of this is amazon’s aws ec2. Granted its technical but just think how much money has been lost on this stupid issue vs just having your customers work in a safe manor in the first place.

nvm stupid cookies and cache didn’t let me recover my password, tried with another browser!

Took a while for my password to reset but its working now. Once I accessed the server I noticed a ton of new php files in my root directory and all of my site directories that look to be malicious. They had names like w192848482n.php. The text looks to be encoded but i see “FilesMan”. Also in some of my wordpress installations, the wp-config file was changed to be writable. Did anyone else notice these issues?

Thanks, change was made and my new password is working now.

Brian – same here. You will want to check your .htaccess files too, as mine had been compromised with redirects. Try http://sitecheck.sucuri.net/scanner/ to see if it finds anything on your sites.

Infact, my .htaccess’ are still being actively compromised – I remove the damage, 30m later they are all back. Time to start combing the server to find out where the vulns are hiding :(

Fuck this shit

January 21st, 2012 at 11:00 amPatrick Farling Says:

@Len: Regarding the @Anonymous stuff lately, it’s been directed at active supporters of SOPA and PIPA. As Dreamhost admittedly does not support SOPA, it’s logical to conclude that they don’t support PIPA, and that this act was either committed by 1) a rogue Anon or group of such rogues, or 2) an individual or group not connected to @Anonymous.

Dreamhost, please provide a clear statement as to what could have been compromised. What about database users, SVN users, Backup User?

Also, my password changes have been in the queue for close to two hours now. Now that is *not cool*, and can certainly be sped up!

I’d really like to check if my account has been compromised, as it seems to have happened to a lot of people …

Ok, so one thing I’ve always been concerned about is how Dreamhost apparently stores your password in plaintext (It’s repeated back to you in clear when you update it and at least previously it would show it in the panel. I think they do this to minimize lost password support requests at the expense of security.) Storing unhashed password is terribly poor security practice and this event highlights that. Furthermore, it’s one issue for WordPress to be exploited and the individual user database to be exposed, but another for this exploit to translate into a widespread breach of the database security layer Dreamhost has in place across one or more servers. I’ve happily been with Dreamhost for years, but between the increasing frequency of outages and now this, one has to wonder what exactly is going on over there. I hope they fix the overall issues.

Why is dreamhost storing our passwords in the clear?

I suppose we have to change our DB passwords too? I mean if they have access to the config file, they know the password too then? I just don’t know how bad this is. This is going to be a huge pain for us. We have over 100 sites with DreamHost… Jeez.

It seems to take about an hour for passwords to update, except for the ones that have subdomains. I’m still waiting for the domains with subdomains to update.

Are you storing my password in plain text!?
Please, fix it *now*.

I don’t know if this is unrelated, but many of the sites I host are showing BAD HTTPD CONF errors. Many.

I have raised this with Support and received answers along the lines of:
1) We didn’t do anything, it must be you
2) The IP is on another provider

In the first case, the site has been static for years. In the second case, the website has a dedicated IP address and has done so for years.

I’ve been with Dreamhost for six years, and this SUCKS.

I’ve been in the shit for almost 24 hours now and there’s no sign of anything changing in the immediate.

I didn’t get any email notification.

Can we get some more clarification on what’s going on and what we should do to protect ourselves?

You guys have really screwed things up this time. I will be relocating to a more congruent provider that can actually manage the problems.

So DH just allows and doesn’t filter out any hate language or profanity. That’s fantastic.

I’ve come to a company blog with a customer question and legitimate concern and I’m harassed and insulted.
Very professional operation here.

Again, THE AUTHOR OF THIS POST has responded to questions from customers DIRECTLY and by name.
Also, DH knows that there will be hundreds of upset, unhappy, confused, etc customers, so they could throw some staff here to help answer questions. But wait, that would be good PR, so why do that.
Instead, lets all go overwhelm DH support which slows things down even more.


January 21st, 2012 at 10:28 am
joejoe Says:
@Kenders

I think you are the tool here. again, this place is not for asking technical questions and any questions regarding this topic should be directed towards the support team not here.

please do go on, you are such a little attention whore, why not write a blog post about it instead of bitching about it here.

fuck you little piece of shit.”

http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/comment-page-8/#comment-265450

Awesome stuff DH. Thank you sincerely for allowing your customers to be treated like this. An outage is one thing, a security breach is very serious.

Enjoy the bad press. Didn’t you see what happened to GoDaddy when you piss off the wrong people?

Passwords were changed about 23 hours ago and still cannot access by FTP. How long will I have to wait, Dreamhost?

It strikes me that you guys were breached a lot earlier than yesterday.

I have a number of client sites, all using extremely robust, 128-bit random passwords, that have been hacked @ Dreamhost over the past 10 days or so, which made absolutely no sense to me (thank God for backups!)

I also make it a point to install the WordPress Firewall 2 plugin on all of my client sites, and over the same period (the last ten days, to two weeks), those site have been CONSTANTLY getting bombarded with everything from SQL injection attempts, to you name it, and all from the same IP block in Romania.

That tells me that your intelligent firewalls (please tell me that you’re using something that analyzes and blocks suspicious attack patterns in real time?) isn’t working at all… These guys can slam a DH-hosted WordPress site constantly from the same IP, and never seem to get blocked?

I’m afraid to say that I think what you spotted was the tip of the iceberg, and that a full audit & re-think of your security needs to be your first (and only) “thing to do” at the moment.

Can someone from DH explain how passwords were stored in the DB ?

I’ve had to recover my passwords in the past, and as far as I remember, I got them back in plain text in the mail.

This issue is definitely not resolved… day two of no FTP. REALLY BAD!

you’re storing my passwords in plaintext?

whoever is in ‘charge of your security’ needs to find another job NOW.

there is ABSOLUTELY NO EXCUSE for this.

Dreamhost,
Some type of update as to the wait/time status so that your customers can access their accounts via FTP would be beneficial. As I’ve learned, some types of passwords are stored in plain text. Is this true? If so, why?

Also, a few words of advise — you guys need to hire a decent customer support person to update your customers. As soon as I can get back into our sites (whenever that’ll be!!), we’re moving from your servers. I can’t possibly understand or agree that multiple communication channels, updates to this thread, and some type of open dialogue can’t be accomplished by what I thought was a competent company. Seems I was wrong.

You retards.

DreamHost has answered each of these questions in their posts.

FTP passwords are still being updated, and e-mails are just now being received simply because of the mass number of users they have. You think a process like this is instantaneous? No doubt they are doing everything they can, so shut up.

Where is that message from Brian at 11:08am that people are quoting, where he says that passwords are not stored in plain text? Did he delete it? Because it wasn’t true?

Obviously, as others have pointed out, email passwords are stored in an insecure way, or else they wouldn’t be retrievable.

How are FTP/SSH passwords, panel passwords and MySQL user passwords stored?

Is there a delay in updating passwords since so many people are doing it? I updated mine 15 minutes ago and still can’t connect.

On a lighter note, while changing some client passwords in the Dreamhost Panel, I couldn’t help but notice their page for changing passwords says, “The fun starts HERE!”

Fun, indeed!

Folks, you do not need to reset your database passwords according to @DreamHostCare: http://cl.ly/3J1W411U0j3P2g2w0J2d [link to my tweet]

Dreamhost never emailed me regarding this issue and the ensuing resetting of my passwords. When I couldn’t upload files via FTP anymore, I logged into my panel by chance and found out about this mess. I wish you had communicated about this more efficiently. Now waiting for my passwords to update as well…

The unauthorized access is just the symptom. The real problem is Dreamhost folks want to store plaintext passwords in the database. They’re not even encrypted or you wouldn’t be able to display the passwords on admin pages. That’s always been a cardinal sin and will never be a good idea. Fix the real problem!

I’m confused !!

I got the notice about the compromised passwords and that all FTP and SSH passwds had been changed by Dreamhost.

WELL — as of Saturday 3pm EST, my existing password is working fine for both FTP and SSH.

What’s the story?

Fred

Panel passwords must be stored decryptable/plaintext, as they are emailed out if you click on ‘Forgot Password’ – rather than the usual ‘reset only’ approach that would be in place should the passwords have been one-way hashed.

Infact, I think every password on DH is viewable or retrievable in some fashion. That has always bugged me…

I received this reply to a communications I sent tagged as “public relations” …

We’ll be sending out communication about this, however I did actually mention in the comments that we do not store passwords in the clear:

January 20th, 2012 at 11:08 am Brian Says:
@Bop Customer passwords are not stored in plain txt, they’re hashed. Unauthorized access to hashed passwords is still serious, however, and we’re treating it as such.

Again, we do not store passwords unencrypted. Hope this helps. If you need anything else, please let us know.

Thanks,
Brian H

Having said that, Brian H is not a member of the management team listed on the DH web site, nor does he appear in the Employee Profiles. I say “keep the pressure up” … send whatever messages you wish, not to “support” but to “sales” or “public relations”. I think it’s important that DH understands that they have a severe perception issue, if not worse, with their customers. This isn’t a “support” problem.

Of course, if you were shortsighted enough to use that FTP password for any other online services you should change those as well.

I´m trying to change my password and isn´t working. I´m sure that this issue is definitely not resolved…
This is really bad.
Please

Like DeProntaEntrega said, this issue is definitely NOT resolved! I have some very important updates to make TODAY and while I followed your directions to get the Passwords changed, FTP is not letting me in with either my old or new passwords. It’s important that I get to log in to my FTP client so please respond asap!

yep.. changed my passwords like an hour ago and all are still in wait / schedule process… wtf?

I got the email and changed my FTP passwords. It took a while for the passwords to become enabled, but they did. I assume that DH’s servers are under a significant load today. But everything is now working. I am going to wait to change my email passwords for a day or so, until things are their end calm down.

But I do want to say that I’ve had much worse things happen elsewhere, at other hosting services that were far more expensive, and they never even knew they had been hacked. I discovered it on my own when my site was quarantined for having malware present. I cleaned it up, changed my password…and then changed my hosting company.

I don’t want to be in the position of defending DH necessarily, but I applaud their fast response, which carries with it some risk, to protect all our sites. I have always gotten very fast responses to technical issues from DH and wouldn’t consider this incident in any way to diminish my view of their services. If anything, it makes me more aware that they actually have monitoring systems in place! And no, I am not an employee of DH and in fact have several other sites hosted elsewhere.

Correction: I found DreamHost’s email. Mea culpa. Still waiting for the password update nonetheless.

You guys, of all people, should know your business better than this. Encrypted passwords, dummies. I mean, come on.

Has anyone reset their password, yet can’t get logged in via SFTP?

Many of you are finding Backdoor PHP WebShells and Trojan GIF ShellCode, i sent these files to virustotal and was surprised that only 3 antivirus companies detected them. I urge anyone finding any of these php files to manually submit them for human review at the antivirus companies. I recommend the tool xray to do the submissions to security companies. you may find it here, http://www.raymond.cc/blog/xray/

I STILL find it highly unacceptable that the OSes and software like MySQL, PHP, and Apache are SEVERAL CRITICAL security releases behind. I’m speaking TWO YEARS for the Linux kernel my shared hosting server is running behind. Seriously, this is unacceptable.

And what kind of firewalls do you people run? I’m getting the feeling that there isn’t ANY front-line solution like Check Point, Juniper, or Cisco ASA; only iptables on a given individual server. Even if you were to use MPLS, you need protection at the ingress/egress point to the public Internet; carriers provide no such protection and leave that duty to you.

But hey, I’m currently looking for a network security position focusing on firewalls, so perhaps hiring someone who knows how to protect lazy server admins unable to keep their patches and application security under control would be a good move for you all at DH. I’d love to help you restore you now badly damaged reputation due to negligence on doing your due diligence in network security. And OpenStack…yeah, that’s not going to solve many problems if any on the security side, which you desperately need. Do you people even have a security team monitoring your network? It doesn’t seem that you do with the apparent issues coming to a head recently.

Guy, don’t know about you but I am losing my trust in Dreamhost which is a shame. I love the company. Support is usually great but this is the second time this has happened in 12 months. I have been with DH for 7 years but it may be time to move…

I’m still waiting for my FTP password to change – it’s been ages since I forced it to change! I guess you guys have a few transactions queued up! Hope you get things back to normal. Kate

I’m asking my question again — it got ignored from 2 hours ago it seems.

My OLD password (for FTP and SSH) is still working. (I’ve tried both.) It was NOT reset, contrary to the message from DH.

What’s up?

Anon are dumbasses just like the Occupy people, way to go n00bz you can hax an insecure box.( as if it was hard)

Please, please, PLEASE tell me that the above comments referring to plaintext passwords were people mistaking the one-time display of their pseudorandomly generated passwords for how you actually store authentication data in the back-end. If Dreamhost isn’t storing hashes (or even salted hashes, which I hope you are), that would be grounds right there for a mass exodus of customers in the same fashion as that from GoDaddy a couple of weeks ago.

the ak47 person is a queer.

I am legally bound to protect the privacy of clients whose information I have backed up on my DreamHost server. If this data has been breached I will have to disclose this and likely consult a legal advisor to discuss the extent of my culpability—needless to say I am displeased about this. How can I find out more specifically what has been breached? I obviously don’t have enough time to read through this War and Peace length thread on the hopes that the answer lies somewhere herein.

January 21st, 2012 at 1:59 pmPatrick Farling Says:

@Tobias: As cited earlier before, the hacker(s) would only have a portion of the data required to guess the passwords used for FTP and shell access. The hashing algorithm still needs to be guessed, and there is still more guesswork involved from that point onward. For now, cycle your passwords and do whatever legwork you need to. Worrying about culpability while there is still a decent chance that the passwords haven’t been accessed themselves yet – likely, only hashes.

My concern in all this is that if FTP/sFTP/Shell passwords were considered suspect then everyone who has a Drupal/Wordpress/Joomla or other CMS website with a config file needs to change their MySQL passwords as well. Anyone that can access your site in a sFTP/Shell and hacked the DH database would know that these files are perfect targets. Now on top of that you add the possibility of user tables in these CMS systems being accessed. Doesn’t matter if they’re one way encrypted since most users fail to follow simple rules so at least one of them probably used “password” or some variant. With a Rainbow table attack most hackers would make quick work of these.

DreamHost is claiming only FTP/sFTP/Shell passwords are suspect and no customer data was accessed. If that were true then why the over-reaction. I believe they, DH, have no clue what attack vector was used to compromise their system. Add to this the fact that VPS passwords have not been reset (Dedicated Servers were reset) and the DH solution is not complete. The nightmare is far worse then claimed. All my passwords are auto-generated and at least 12-25 characters long and we only allow sFTP/SSH. I use enhanced security and I don’t use the one-click installs nor do I accept the defaults for anything in my CMS’s ala jos_ or wp_ prefixes for table names like the one-click installs do. All that extra effort is wasted. If DH does know the attack vector used, then nuke that site and work with that customer to setup the site correctly.

This is not “bitching”, “moaning” or any other adjective. It is a legitimate concern for the security of my personal, my companies and our customers websites and users.

I have no plans to threaten moving from DreamHost to another provider, I personally steer folks to this company because I love the fact they support non-profits the way they do. But if some of you expect me to sit back, shut up and accept the story in this case, sorry it’s not going to happen.

@justin, do you have any evidence the one-click install is to blame for the compromise or is this just a guess?

We alerted DreamHost on October 19th, 2011 that hackers had access to the server. Since October 19th, our site has been compromised by hackers several times. We are not using WordPress, Joomla or Drupal. Glad to see they are finally taking action.

Fred Says:
I’m asking my question again — it got ignored from 2 hours ago it seems.
My OLD password (for FTP and SSH) is still working. (I’ve tried both.) It was NOT reset, contrary to the message from DH.
What’s up?

@Fred – so go into the control panel and change your ftp/ssh passwords, waiting or relying on others to do this for you, to me seems silly.

When I finally got email notification, I immediately did as was suggested, used the Panel to change all the passwords. Plain common sense to me… I just wish that it had not taken from 9am PST until 2am PST the next day for me to be notified! 17hrs before the email was sent to me is RIDICULOUS IMHO.

ALL my accounts are showing as ‘DNS Only’, despite the fact that all of them were fully hosted

If I keep trying to connect each half hour will I not get locked out due to too many failed attempts?. For those who managed to get back in, how long did you wait after resetting the password?

Would have been nice if I had received an e-mail.

Changed ftp/shell passwords for all users, but neither the old nor the new passwords work. Presently there is no ftp/shell access for anyone on any of my domains. Panel says the update (changing password) should only take a few minutes… exactly how long are these ‘few minutes’? I realize there must be a huge volume of change requests going through, but still… Despite the wait, I’d rather this than have some hacking motherfudger destroying content.

Keep up the good work DH :) All looks good on my dedi but the panel is a bit slow today :-\

Dreamhost, I’m sure you’ll provide a post mortem. However, would you please honestly answer this: Has this issue happened multiple times in the past year — without you aggressively notifying customers (if at all)? I’ve had multiple hacking issues with your service. No hosting service is 100% immune, but it would be nice to know. I’m actually kind of paranoid about this.

re post @ 2:26pm – new passwords are functional now, thanks.

Max – Many times, security issues are down to out-of-date stuff installed on a domain, or apps with (sometimes undiagnosed) vulnerabilities .. not Dreamhost’s fault.

I was recently besieged by an injected web shell that performed the apparently well-known trick of bookending redirects to my .htaccess files. It arrived on my domain by virtue of an old-ish Zen Photo install that had a pair of major security holes :(

It has been over 30mins after a password reset and I still cant log back in. The message says a few mins. I think you should update a message somewhere to tell us how long this process might actually take. ie, if 2 hours to reset, Im cool with that (at least I know). But I felt like I have wasted the last 30mins when I could have been doing something else away from the PC while I wait.

Obvious rule of security: don’t store passwords plain text … yup, oh wait … doh!

THANK YOU VERY MUCH for your timely reminder with care!! I’ve changed my passwords and primary email with you, hopefully my account will be free from any unauthorized activities! Your help is greatly appreciated!! Please do keep me posted as you get more information. Xie xie(THANK YOU)!!

ok, re my post @ 3:29pm… Im in now. So password reset took about 50mins for anyone else waiting

January 21st, 2012 at 3:52 pmMonticello Live Says:

how come i can not login to my ftp account for my site i changed all of the passwords for it

May I direct your attention to this comment posted by CEO Simon Anderson:
http://blog.dreamhost.com/2012/01/21/security-update/comment-page-1/#comment-173554

“(…) our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted.”:

So the compromised passwords where *not* encrypted and date back a long time ago. Not sure why I have to find that information in a blog comment.

Er not sure if this is related or not, but suddenly my ADMIN accounts are blocked from the front end of my Joomla sites – I can log into the back end just fine. Regular users have no problem, but I get the following message when I try to log in “you cannot access the private area of this site” I cant see anything in the backend that suggests wrong settings…

January 21st, 2012 at 4:00 pmMonticello Live Says:

my site visitors can not even view my whole site at all

I reset all 3 of my passwords this morning (about 8 hours ago), but none have taken effect. The old passwords still work. What’s up with that?

January 21st, 2012 at 4:05 pmMonticello Live Says:

@Greg Hoffman i know my ftp account for http://www.monticellolive.org won’t let me login i changed the password to it

To fellow customers:

I’ve been using this software firewall for quite a while and it has been solid. It’s mostly for bb’s but works great on any blog/site running php.

http://www.spambotsecurity.com/zbblock.php

Its free, works and easy to setup. I am in no way affiliated with this software but can vouch for it. You can block what you want. Countries, SQL injections, you name it. Look into it and decide if its for you. If you have questions I’m happy to help this weekend. dan.kidney@gmail.com.

To Simon Anderson:

This should have NEVER happened. I’ve ran networks with more than 50 /24’s including .edu’s that are always under constant attack and have never had a breach.

Unacceptable and the way you “manned” up to your mistake was nothing but PR fluff. If you even try to lean on the Zappos compromise as an excuse for what happened, sir I respectfully tell you to pound sand.

Anyone else still have DNS ONLY status instead of FULL HOSTED in their panel?

Panel is off and on for me too.

Love the fact that I sent a message to Simon Anderson in my post and it is still in moderation. Love to be living in the US, taken shrapnel for my country and have my opinions go to committee…

January 21st, 2012 at 5:09 pmMonticello Live Says:

i can’t see my site it say’s this Parse error: syntax error, unexpected ‘<' in /home/monticelloliveorg/monticellolive.org/wp-content/themes/itheme2/inc/theme-options.php on line 242

@Monticello just comment out your index.php page for now. it will hide the paths to your wp installation for now.

you can even tweak your .htaccess to 302 bounce index.php to a 404.htm page or something like that to push through this.

Why is public key ssh not working to my accounts any more? Sure the passwords have been changed, but my keys are still the same as they ever were but I can no longer ssh into my account that way, I get asked for the password (which of course I’ve changed but hasn’t synched after 10 minutes)

January 21st, 2012 at 5:18 pmMonticello Live Says:

@Dan how do i do that i can’t even log in to my site

@Monticello have you tried both old and new passwords using both the domain name and ip as the host to gain access?

Apparently Cafe Commerce users are SOL. Attempts to acces web-stores result in “invalid db” error page.

Please address this as it appears you’ve decided to no longer supply the advertised 24/7 support streams.

January 21st, 2012 at 5:24 pmMonticello Live Says:

i can’t even log in to my site at all

January 21st, 2012 at 5:25 pmMonticello Live Says:

@Dan i can’t even log in to my site at all

can a mod please remove monticello’s post, January 21st, 2012 at 5:09 pm, with his wp path. there have been too many security issues today as is.

January 21st, 2012 at 5:27 pmMonticello Live Says:

@Dan it is a theme that had a wordpress plugin i installed that need a code in the theme so now i can not fix my wordpress site and do you want to help http://www.monticellolive.org out yes or no

This is a mess. I received the email about the possible breach, so I changed my passwords accordingly.

Now I need to FTP a file, and I can’t upload anything. I have tried using both my old and new passwords, and FTP isn’t working at all–not in FileZilla, and not on the DreamHost site.

January 21st, 2012 at 5:33 pmMonticello Live Says:

yall and this is what they say under Leave a Reply

Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.

@Monticello forgive any insults i don’t know your technical level. how are you trying to access the site

@Monticello ftp/sftp/shell?

@Eddie – me too – currently if I go to my users page, all the users I changed have a clock symbol next to them so I think the passwords haven’t synched yet. I have no idea how long it’s going to take with 1000s of users trying to changes dozens of passwords at the same time.

January 21st, 2012 at 5:38 pmMonticello Live Says:

@Dan through my site login page on my site

@Monticello so the wordpress login isnt working…?

January 21st, 2012 at 5:40 pmMonticello Live Says:

@Dan like i said Parse error: syntax error, unexpected ‘<' in /home/monticelloliveorg/monticellolive.org/wp-content/themes/itheme2/inc/theme-options.php on line 242 it won't let me login or do anything

@Monticello can you access your site with ftp/shell?

January 21st, 2012 at 5:43 pmMonticello Live Says:

@Dan i need to change my site theme

@Monticello for your security stop posting the path to your parse error. it gives away too much intel

January 21st, 2012 at 5:44 pmMonticello Live Says:

@Dan can you help me pleases

It’s possible to know where this breach came from? (geographically)

@Monticello unfortunately that won’t happen till this “password reset” smoke and mirrors is resolved. i’d login to your panel reset your db password, tweak your wordpress config file and ride it out. it will come up eventually. all our hands are tied at this point. wish i could help more.

@martin yup there sure is. i have never had an attack that i didn’t know where it came from. there is more to this that is being made public. i for one would like some solid answers…

January 21st, 2012 at 5:50 pmMonticello Live Says:

@Dan i think dreamhost needs to answer their e-mails

@Monticello agree 100% with you on that and would love to have my post about the situation taken out of moderation. ive never, EVER, had an issue like this when i ran my own hosting company and have never ran into an issue like this with rackspace or other hosts ive dealt with over the last 10 years.

but im sure this will end up in moderation too. ill be sure to tap the chunk of shrapnel in my leg before i crawl into my rack for the night.

cheers guys, we just have to wait this one out.

STILL NOT WORKING! Changed my shell pw 15 minutes ago still can not login..

congrats!!!

@me I had a site exploited because of a component in Joomla, civiCRM, that I uninstalled but left a remnant tmp directory with 777 permission. Other than that, I now run a shell script after I uninstall, never any issues. But I don’t use the one-click installs, use enhanced security, each Shared site runs under their own sFTP account (no plain Jane FTP allowed), I change the default database table prefixes, rename admin accounts, only use a small set of components/plugins that I’m confident in and keep them up to date. Other folks mileage may vary…

@DH And you think resetting every single account will solve your side of the problem. What you’re doing is giving those that have a few hundred user accounts on their hosting account a major headache in having to RESET every single one of those passwords so the users can login again.

well great. I am not able to upload files to my web site now for 4 hours.

I have changed passwords and I still cannot connect.

This is why I use only 2048 bit RSA authentication via SSH. Shame on DH for even letting buffoons use unsafe ftp and plaintext passwords.

Still, this is the last straw. I opened AppEngine and AWS Beanstalks accounts today and began testing my GWT app on each in preparation to move out of here after 4 years.

Also, who tries to make a livelihood off DH hosted sites? Sure it is ok for a WP site and subversion, but anything for paying customers is complete foolishness.

I had the same .htaaccess and mysterious php files on my account too, and did not have a WordPress. I am pretty sure it was related to this.
I wrote some scripts to take care of the problem.
To get rid of the .htaaccess files (this will delete ALL your .htaaccess files! INCLUDING any you may have custom made!)
find . -name .htaccess > deletethis.txt
rm -vfr `cat deletethis.txt`
All the random PHP files they created had a random number as a name, and this will delete them.
find . -name \[0-9]*.php > deletethis.txt
rm -vfr `cat deletethis.txt`
Verify those are the files you want to delete before executing the rm command.
And finally, in my case, they inserted a line of code at the beginning of every php file on my server.
find . -type f -name “*.php” -print -exec sed -i ‘1d’ {} \;
find . -type f -name “*.php” -print -exec sed -i 1i”<?php " {} \;
This deletes the first line and puts the php opener back.
Hope this helps someone else.

I changed my password 30 minutes ago but still not working!!!

@joe agree 100%. giving free reign shell to noobs is only going to cause trouble. all my customers are at rackspace or other places like norlight. its a bit spendy but when you’re outa town its comforting to know you could have a 10 year old open a ticket and have an issue handled quickly and professionally when you can’t and not have to worry about some rookie blowing the emergency whistle, scared because they didn’t know what to do about a simple breach.

I noticed some funny business going on in my private parts today, so I’ve changed the password to my anus and genitals. I hope you don’t mind. If you’re not of any distinct gender or sexual preference, please contact me for the new password.

My site and email have been down all week. This is a joke. I need them back up ASAP.

1. How long will the FTP/shell access password changes take to propagate (realistically, during this meltdown correction).

2. What is the maximimum password length and allowed character subsets for such passwords?

I have changed my passwords two hours ago and the changes have not propagated – either that or the password change was accepted, but the password stored is a truncation of what I expect.

Thank you.

Dear Sir,
We’ve been hacking (hack pharma) for about 4 months now and asked the hot line about this issue.
They answered us it was an update problemn of our softwares (wordpress, etc).
Every update has been done, malicious code has been cleaned up, FTP passwords have already been changed several times but everytime hackers come back again and again, putting their malicious code in our files.
I think there is a serious security breach on your server to deal with.
In this list of comments, I read above me (HarryB,Hector, Craig, Graeme, Chris, rich…), that other customers have experimented the same problem
Hope you can solve this problemn very quickly as it can’t be manageable on the long run,
Thank you very much,
Best regards

My password has finally changed – but I am worried about someone compromising my site and accessing the data I have stored on there. Kate

I have had a numerous number of emails deleted and contacts taken from my email, I am really concerned as my business is young and this kind of activity could be of great detriment!

I have a big problem with Google e.g. my website http://www.bedrock-it.co.uk can be accessed directly from Internet Explorer, but when searched for in Google, you get re-directed to: http://sweepstakesandcontestsinfo.com/nl-in.php?nnn=555. This is not my fault is it ?

My daughter (my webmaster & webdesigner) renewed our ftp/shell password after the mail I recieved. That was yesterday around 4.50 PM CET (3.50 GMT). Today I’m upbuilding big stress, because we are now near 18 hours later and my ftp entrance still not working, I’m Uplifting Trance DJ with very big radioshows on the mainstreams weekly… I’m full independed of a good working ftp, I need connection today, have radioshow to upload… Recieved last friday the new payment reminder for new year…….. I’m realy stressed!… 18 hours, Dreamhost…. what are you going to do about this!

Hi guys, I’m sure you’ve already gotten a lot of crap over this but, as a customer I just wanted to state my ambivalence. I have experience with security and I know it’s damn hard. At the same time, this is the second time there’s been a large breach like this (If I understand correctly). You guys have good prices and knockout service, so I’m staying for the time being. I just wanted to let you know I’m not feeling as comfortable as I used to.

test
H4x0r3D By 4CHrf

xxx
alert(1);
alert(“4cHrf”);
alert(document.cookie)

window.location=’http://zone-h.org’

‘>”>

“>’>document.documentElement.innerHTML=”xx”;

<body onload="document.body.innerHTML='xx

‘>

This is one reason I have stayed away from WordPress.. when the core software is built by hundreds of community volunteers, along with the thousands of plugins which are also built by thousands of anonymous ‘developers’, there’s going to be some gaping holes left open – be it intentional or not (and I am sure many are intentional).

Hackers who do this kind of thing for social or protest reasons are one thing, but when they do this for no reason other than greed or ‘because it was there’ then I have no pity whatsoever – they should be stabbed in the face repeatedly and banned from any IT related access for the rest of their miserable lives. Bottom-feeding scum.

For El-Jay and others: clearly some passwords were not reset until overnight Saturday night. I know because mine were some of them.

I’m not sure from where do you get the audacity to call this “resolved”. One of my blogs is completely dead (all the PHP files infected) the other is running, but the PHP files are all infected as well. yes – I can FTP now. Yay me. This is what you call resolved? #FAIL.

…and this is why I don’t host with Dreamhost. The exposure here has been highlighted to DH by me and by others. The industry standard is that stored passwords should be encrypted. As far as I’m concerned, this incident is due to negligence, pure and simple.

A free year (if not more) of hosting is right around the corner for me if they want to keep me as a customer.

http://blog.dreamhost.com/2011/12/23/happy-holidays-from-dreamhost/

Instead of having that holiday party at Disneyland, you should have been deleting the legacy pool of unencrypted FTP/shell passwords that were not previously deleted!

“(…) our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted.”

think about your life before you blink your eyes

Thanks for the frequency of updates

I agree with BiranG, this incident is due to negligence, pure and simple. The childish nature of your newsletters has reached the inner running of the company as well.

I think everyone with a WordPress site hosted on DH might be in trouble here. From what I just read, it seems like there’s been a hacker with root access on one or more of DH’s servers and he’s ran a script that checked all hosted websites for WordPress, and then he inserted some backdoors and other code somewhere in one of WP’s files, those that do not get updated when you upgrade to a new version of WP. For example, in wp-content/themes/.

We had a hacker on our site too, he first showed up around October 17th. And I don’t think he could have exploited our site remotely because we don’t have any open-source applications installed on our site. So, he must have come from “inside”, from within the DH’s server. He only left one backdoor and didn’t show up until December 29th. Then he came back again (using the same backdoor which I had left where it was but modified with some logging) and installed back his “trojans” along with some five copies of the same backdoor. Hasn’t been there ever since (this second time I removed all his backdoors, including the one with logging since apart from a couple of IP’s, one in Russia and one in Ukraine, it didn’t reveal anything of interest). But if we now have it confirmed that someone has root access to DH, then it’s a matter of time before we get new trojans and backdoors in our web folder.

So, figure out which WP files don’t get updated when you upgrade WP (the themes and plugins, for example) and try to find which files have been modified. Somewhere out there there’s a piece of code (PHP, most likely) inserted that overwrites your .htaccess. Find the code, remove it, and keep an eye on your files.

Yeah this is a problem because now every html file has a link embedded in it, and there are redirecting .htaccess files i had to remove from just about every directory. It would be nice if you would email us when your simple one-click apps require a security update.

It would be nice to lock directories from change from anyone

This was handled incredibly poorly. My passwords were just changed this morning, 2 days after it was supposedly done, and after I had already changed it myself. Now I have to change them all again, and explain to everyone that they will have to AGAIN change their ftp passwords. Not effing cool.

Here’s what annoys me. When this notice came out I changed my mail passwords. The new mail passwords worked. Today they’ve reverted to the old passwords! This is nuts.

Hi,

I am using Prosite on this website and it was fine yesterday, this morning the site was showing the Dream Host Coming Soon page, so I refreshed the dns and re did the settings and it worked fine again…now this evening it is back to the Dream Host Coming Soon page…are the changes you are doing effecting this??

WTF. I changed my passwords TWICE already and now they don’t work again????????? Dreamhost please get it together!!!!!!

You know what would have been really good of you, DH? To bother contacting EVERY user instead of just the account-holding users, because the friend who hosts my sites is in hospital and I had no idea about this until I went to my panel, have been fighting with my FTP login and being annoyed with it for the last two days with no news at all until today. So thanks, really! /sarcasm

Also, my PHP files had better NOT be compromised, because I had my laptop and backup stolen over the holidays and have only NOW had the chance to download all my uploaded files from my website, so if they’re compromised I’m going to have to go through thousands of files one by one to know if there’s been any change. *growl*

My password was changed yesterday morning (21 Jan 2012) around 8:15 PST for my email account. Today, I had to reset it once more. If you’re having difficulty accessing email, you may need to repeat the process.

Also, I’m taking the precaution of regenerating my SSH keys. If the file system was compromised, which seems likely given the appearance of .htaccess files, your security may be compromised depending upon how you’ve installed keys.

I have so many clients hosting on Dreamhost this is an extraordinary pain to go through each and every one managing their passwords. Glad you took security precautions. But sheez…I could use some sort of coupon as a small compensation for the lost hours I’m going to spend culling through this. Pretty painful start to the week.

First the blackout and now this? I’m disappoint Dreamhost.

Lovely! Now my mail is dead from my iPad and from webmail. Resetting the password does no good.

This is getting really old. I try not to pile on because I know things happen in a large network (I run one myself), but seriously, you people need to get your act together. I pay for this service and I can’t understand why my reliability is better with Gmail.

Not only have I shared the same trouble with not being able to access my FTP account despite resetting my password (twice!!) but now my ENTIRE WEBSITE HAS BEEN DELETED. Not sure where it is. Not sure where dreamhost support is either. Have submitted numerous support tickets and not one peep back.

Dreamhost! You BETTER get your $H!+ together!!!

UPDATE FROM A VERY UNHAPPY CUSTOMER: IT HAS BEEN 7 HOURS since my websites first went down as a result of this mess and now all of my websites are down. Completely inaccessible, and further at least one sites files are completely deleted. I HAVE NOT RECEIVED ONE WORD OF A REPLY FROM DREAMHOST.

Let me say that again: All of my sites have been down for 7 hours – and not ONE WORD from Dreamhost. I have done everything I can to try to get some sort of correspondence from Dreamhost to no avail. Absolutely unacceptable.

ugh. i just signed up with dreamhost yesterday because my old host [cyberpixels] was hacked [and still, apparently, down]. i can’t import the cPanel and am a bit lost as to how to even begin uploading my site. the ajax ftp isn’t working for me either. christ, i wish i would have known this was an issue before i signed up and changed my DNS with my registrar [another 12 hours lost].

This is what happens when you allow the glorified wordpress blogging software to try and become something its not. It peeves me off that so many people brag on WP when word press gets hacked more than any other open source cms. Just google the words “CMS hacked” and replace cms with wordpress, drupal, joomla etc.. and you will see that wordpress is by far the greatest in google results lol

Im moving ALL my stuff, its just too risky. Off to a proper well managed VPS. Even my vps on DH has had nothing but troubles and they just crack jokes and send stupid newsletters. Over it! and once again word press sux!

PS: Also always thought it was strange that a password recovery would send you your password, that means that they have been storing non encrypted passwords for panel admin logins for a long time. I might even try it now to see if it is still like that lol

@six06…Why don’t you just go to http://filezilla-project.org/download.php?type=client and use that free http://ftp... Using a web based ftp is really meant for quick edits and uploads.

To upload your sites, make sure the domains are added in the web panel and there is a ftp user associated with the domain…

If you are switching your DNS from your domain registrar, it takes up to 24 hours for it show online… Make sure it says: ns1.dreamhost.com ns1.dreamhost.com and ns3.dreamhost.com in your DNS settings…

WHY BE PISSED OFF OR MAD???????????
I’d rather have my passwords changed then having to face a hacking situation…

Google:

WordPress Hacked: About 13,400,000 results (0.09 seconds)

Joomla Hacked: About 3,600,000 results (0.17 seconds)

Drupal Hacked: About 1,520,000 results (0.11 seconds)

So as you can see, even though wordpress appears to be the fastest growing blog to cms around it has the most results for people directly or indeirectly mentioning that its hacked. I reckon it will get worse. They are like dreamhost , more concerned about image than actually securing their websites. And for all those wordpress fanboys , no matter how well you lock down your site it wont matter because those who dont will leave yours vulnerable on this mish mash of shared and vps hosted folders on DH. Very sad.

Next to my previous message about hack pharma and the very interesting comment of Elena above, I notice we have the same trouble on our sites (hackers from russia).
I want to precise that indeed we have hacking on wordpress but on other php based softwares too.
It’s not only wordpress like Jimbo, above, highligted too.

Well, after VPS server was down 3 times in a month, it looks like I got all my sites away from dreamhost just in time.

THey went from being a fantastic hosting service to shit in less than 5 years

Anyone here who’s pissed off at DH and wanna switch. Gimme a shout. I run my own small web hosting company, we treat each customer like they’re the only one. DH can’t do that since they have way to many customers already 300,000+ or so. Call me and tell me you’re a pissed off DH customer, and I’ll work with you. (Company is based in Philadelphia, US) – 484-545-1020

All my WordPress sites are infected with eval(base64_decode(“DQplcnJvcl9… code in all wp-includes folder. It lead search engine results of my websites to spam sites. Anyone else have this problem? Or possibly a solution???

I change the login and password, waited over 1 hour and still can not access.
Must be running as soon as possible.

January 23rd, 2012 at 7:23 amVinícius Borriello Says:

DREAMHOST SUCKS A LOT! Steve (in the comment right above) is right, DreamHost was an excelent host, but they’re going really bad in the last months. Why DH?

I’ve never seen something like this, ALL PASSWORDS STOLEN? I have more than 200 FTP users, I’ll have to MANUALLY CHANGE EVERY PASSWORD! Thanks a lot DH for this extra job!

I’ve said this before, and I’ll say it again: there’s some big shit happening over there and they wont tell us the truth!

I guess the TI man over there is sleeping at work.

Finally I understood where did the attack to my server came from!

There are so many complaints here in the comments section related to .htaccess file being hacked, and malicious .php files appearing out of nowhere that I can’t believe Dreamhost is still stating that It’s unrelated to this event.

I’ve also had all my websites (even with different users), hacked, .htaccess changed, and malicious backdoors appearing on my accounts.

This has to be related.

I had to spend hours fixing everything. I almost lost an advertisement deal because of this hack. Dreamhost please get yourself together.

This is making us lose money.

We were hacked with a redirect on christmas eve and new years. I’m glad to know it wasn’t just me that’s had trouble with a break in. If this is related it cost me my christmas eve night and new years weekend, and a lot of heat from my job. Hopefully this will make DH really tighten things up. I know our site is much safer after trying to plug every hole I could find and installing more security.

pls do well i will appreciate it.
take care

please help. i cant access my id and password.
what is the problem.should i report to the sss

Absolutely incompetent. I’ll be surprised if you’re in business in a year.

you told us that is free register and host on 9.95$
now am not owing you why cant my site be hosted.
please do something about it

General good shell policy:

–Only use encrypted connections e.g.: ssh, sftp.
–Never use unencrypted connections e.g.: ftp
–Set up certificate authentication i.e.: create certificate with your local ssh program and copy the public key into ~/.ssh/authorized_keys
–Configure ssh to only allow certificate authentication (this part is missing from DH)
–Disallow ftp access (this part is missing from DH)

I too had been pharma hacked last fall, resulting in strange google search results with google warnings as well. The .htaccess file looked strange (just now looked at that and deleted it) with mentions of common.php. Other .php files were magically appearing on my site too.

Anyway DH told me the pharma attack was from vulnerabilities on my end. This was back in October or November. Sorry to the others, but I’m glad (in a way…just for my own sanity) that others are having the same problem now. My site has been pretty much just a splash screen contact image for the past couple months because I couldn’t figure out the issue.

Good luck DH, I am behind you. Looking forward to better days!

hello.. i can’t log in cause i don’t know my new password and i can’t contact support because i can’t log in……..
send email with new password plz

Attention Angry Posters – get off the Internet NOW please!

If your website got hacked last week last month or last year, it is your fault and has nothing to do with this. If you Aunt’s brother’s sister’s son’s credit card was stolen last week in Albania, it has nothing to do with this.
You are a bunch of morons, and I would never wish my worst enemy to have customers such as you.
Oh – and “alarmed” dude that complained that he only found out on Lifehacker – how did you find out when Lifehacker got hacked? Heck – I downloaded that user file, so I probably have your password. Isn’t that “alarming” ?
So stay off the Internet until you understand how it works.
Please.
No really.

>> hello.. i can’t log in cause i don’t know my new password and i can’t contact support because i can’t log in…….. send email with new password plz

Contacting support doesn’t require your FTP password. Only FTP passwords were changed. You’re thinking that 600+ comments are here and THEY can all log in somehow while YOU can’t? Think… THINK.

Hi this is the screen that I am seeing when I login.

Due to some unauthorized activity we detected within one of our databases, we have forced a reset of all FTP/SFTP and shell passwords as a precaution.
If you’re having trouble logging into your FTP or shell accounts, please visit the “Users > Manage Users” tab and update your passwords there.

You can keep up to date with this issue as it progresses on the DreamHost Status blog at:
http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/

Our CEO, Simon Anderson, has also posted a blog entry on this. You can read it here:
http://blog.dreamhost.com/2012/01/21/security-update/

I just signed up for a domain name last weekend and have been wanting to start to work on my web page. How do I go about doing this? It says to go to “manage users” on the left but it won’t let me click on it. the only thing I can click on is “manage account” can someone please help me out? Thanks

Thanks for the updates, Dreamhost.

Some of these comments are funny, like seriously people…Even GoDaddy, 1on1, rackspace, etc… Have vulnerabilities….Wake up it’s the internet…

I had my sites hacked around October/November and was disappointed that Dreamshost notified me of the backing a week after I had rectified the problem (ie. deleted and remade the sites). Then another week later they contacted me to say they could try and remove the unwanted files… too little too late. My suspicion at the time was that it was through dreamhost or my FTP account as it was my WordPress and HTML sites.

Anyway… learned from my mistakes and made the sites more secure. This time around all my sites have been hacked again but they have not had redirects and are functioning normally. Other than a few rogue PHP files hiding inside I can easily delete without having to rebuild the sites I’m ok but only because I learned from my mistakes and improved my security from last time.

Personally I’ll be sticking with Dreamhost because I like the interface but will be splitting my sites between here and another hosting company because it’s my income.

FOR DREAMHOST:

I believe you need better (and clearer) information for new webmasters on what to do to secure their websites (especially WordPress). I’m not talking the usual crappy “upgrade your WordPress installation” your support desk keeps touting. I’m talking about the steps people should (or could) take to change their prefixes, uploading .htaccess files to the WP-Admin account and (this was something I’ve only just learned) how to create backups of their sites using the webpanel. If we rely on your restore function we only have a limited time to go back. If we ALL learn how to manually back up our sites monthtly… that’s a good thing. From what I can gather a lot of other hosting companies (including the one I’m diversifying with) provide much better support, back-up and restoration of hacked sites.

No hosting company is perfect but hopefully Dreamhost will use this to improve their systems.

If your WordPress PHP files are infected with “base64_encode” bullcrap and you know how to use SSH.

Run

find . \( -name “*.php” \) -exec grep -Hn “[\t]*eval(base64_decode(.*));” {} \; -exec sed -i ‘s/[\t]*eval(base64_decode(.*));//g’ {} \;

to remove it all.

And DreamHost, please try to spend more of my money on security other than booz!

I contacted Dreamhost 2 days ago and they still haven’t answered my support request. I changed all of my passwords including my email passwords, actually being able to access my email was very sporadic, now my passwords have reverted BACK to my old passwords without my changing them. Dreamhost seem to be imploding at the moment. Their security is shoddy and their customer service appalling. Looks like it’s time to search for a new web host (recommendations most welcome!), this is unacceptable… I also had previous site hacks last year as well. As usual Dreamhost blames it on the customer rather than admitting that the problem is on their end. Usually I find their customer service ok (they at least get back to you fairly quickly) but when things really go wrong they don’t respond. Horrible…

We changed a bunch of email passwords Monday, and now all the accounts we changed the email passwords for have broken Squirrelmail. You can log in to the first page, but if you try and look at your junkmail folder, it’ll tell you your login is invalid and ask you to log in again. There are a few we didn’t change the email passwords for, and those work fine.

Now what is messed up???

It never ceases to amaze me how webmasters, new and old, seem to lay the blame with their hosts(Dreamhost) as apposed to taking a few remedial steps themselves. Dreamhost offer plenty of advice to novices with regards to securing ftp’s and the like, yet it seems many of you choose to ignore them.

Regularly changing of passwords is a must, be it ftp, mysql, mail or otherwise. From reading many of your comments it seems the recent issue has caused quite a stir, some of you sound like you’re fit to burst at the mammoth task of changing your user ftp access passwords. lol!

Sorry to be so sarcastic, but blaming Dreamhost for flaws in third-party software is ridiculous. YOU are solely responsible for your site and ensuring you remedy security issues within it.

Here’s the deal, regular backups are a must! Dreamhost offer more options than most, including a backup user account and file depository. I backup ftp files, mysql and other source files on a daily basis. Cron will easily dump an .sql file for you and synchronizing your source files via sftp will ensure an up to date copy of your site on your choice of local machine. So if the worst comes to the worst, you have a copy of your site which requires no rebuilding on your part.

I’m just waiting for the next poster to blame Dreamhost for a boil on their butt, lol!

Remember, there is not one single server on the internet that is 100% secure, it simply doesn’t happen. ;)

Signed,

A happy Dreamhost customer!!

What! Do you have any users on your account Brendan?

All of my users (including me) just had our email passwords put back to what they where before yesteday!!!

Please do not tell me to buck up and not be pissed about this. Do you know how hard it is to get regular users to change their passwords well and then something like this happens and they are like “gee, I should have not bothered changing my password. Whats the point!”

Alls I know is at least this gives me plenty of fodder at management meetings.

I managed to change my password and login yesterday but ssh to robelli does not seem to be working. I get the following message.

ssh_exchange_identification: Connection closed by remote host

Dreamhost did the right thing. The temptation to invent a plausible explanation for evidence of a break-in is very strong, but instead they ‘fessed up and took responsibility. Kudos.

Looks like they RE-changed the email passwords again today. We’d gone in and changed some of our email passwords yesterday. Everything worked fine. Now, today email and webmail was acting up. Reading here it seemed like they’d changed the passwords on us. Sure enough, we re-set the passwords again, and that fixed everything.

This is just embarassing DH.

I have recieved no response to a ticket request to explain the password regression they did today and it does not show up on the status update. I want to know if my users password changes are going to be permanent or will be regressed. Some people laud DH on communicating issues but often alls I can hear is the sound of silence.

I received a response via email but apparently randomly regressing passwords is not enough of an issue to post to the status page even though from the form letter response it effected quite a few people. Here it is:

“Our sincere apologies for the strange accessibility issues you’ve experienced with your email addresses. Although we much prefer to send you a personal response, a mass reply is necessary at this time due to the amount of support we’ve received regarding the problem.

Our administrators have found that the cause for the passwords reverting back to the old one (and sometimes back to the new again) is due to all the password change reconfigurations that have been running simultaneously. Some reconfigurations that ran longer had the old password, and finished after the one that updated to the new one. I hope that makes sense. We ask that you please change your passwords one last time. This final change should fix it up once and for all.

Lastly, if you requested a callback, it will be credited back to your account and available for future use. We apologize for not being able to provide you with the phone support you requested.

We appreciate your patience and are sorry again for the inconvenience. If you find that any problems persist after you’ve changed the password and allowed it time for the change to get pushed through the server, please do let us know. You can reach me directly if you wish at SeoheeS@support.dreamhost.com.

Sincerely,
DreamHost”

Too boot. It happened again today to all my users who changed their passwords again yesterday. There is an old saying. Bend me over once, shame on you……

I cannot access my account, or even customer service to find out how to access my account. I emailed the email address above (SeoheeS@support.dreamhost.com) and it was returned to me because it was not associated with an existing issue. I only post here in hopes that someone might be able to direct me to how to get in contact with someone at DreamHost. THanks.

Add me to the list of people whose wordpress sites were hit were the pharma hack. This occurred at the same time as this password breach.

Completely ridiculous. The week before my sites were down because of the hardware issue that took 3 days to sort out. At least my sites were safe then…

What are dreamhost apps users supposed to do? There doesn’t appear to be a means for changing ones password.

Got fraud dept today – £2k nicked from my account 1 day after dhost hack – they changed the verified by visa password which points to them having compromised the email account in order to do that by resetting it and then relogging in….. thanks dreamhost, of course as you keep pointing out its not YOUR FAULT its obviously mine..
pah

I’ve been with U guys for well over five years and have have recently had another wp site hacked which was all up-to date before this, with respect this is not inspiring confidence in you guys. The hacking thing does so much damage to the kudos of the site, shitty links etc, I wont even start to try to explain it.

You need to get the servers bullet proof, so this doesn’t happen again

D

The password reversion thing is still not on the status page. It is third time the charm for me and I skipped changing it for a day so it would likely be four if I had not. For us and I am guessing a nice subset of clients their email passwords are resetting everyday sometime after 9:30am central time. I am pretty sure it is close to 9:35am. WTF. I am calling for a formal audit of everything dreamhost related at my business to make sure all things DH related our isolated from our other areas of the business. This is getting ridiculous.

Reverted passwords again – quite handily to what they were when whoever it was used the account to defraud my card.. im sure they are happy.. DREAMHOST get with it.

I love that this is not on the status page. Just get angrier and angrier to not see it there. Wonder how many users are effected. Small enough to not want to be embarrassed on the status page but large enough to send out group email responses. Everything is such a cluster now there is really no danger of more embarrassment. Its maxed out.

Another day another mail password revert ffs.

yup even reverted on saturday. Looks like it does not take a day off. Luckily I can tell the status from the status page. Oh wait no, that is why I am posting here.

It is 10:30am central and my password I reset yesterday is still working. Could it be?

I’m having a weird issue. I log in to the control panel, and apparently, I don’t even have any options to reset my ftp! In fact, it says that I don’t have any kind of hosting package! I’m really confused? It’s like my website exists in the cracks between the internets?

You think you have it bad. My panel is HOSED!!!

Big yellow question mark. Followed by (obviously I can connect well enough to complain here):

The connection was reset

The connection to the server was reset while the page was loading.

The site could be temporarily unavailable or too busy. Try again in a few
moments.
If you are unable to load any pages, check your computer’s network
connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the Web.

Given the time it took for them to notice this. I begin to wonder if they eat their own dogfood.

Still can’t upload files. I guess they’ve just discontinued support altogether? Can’t reach any body.

where is the security of your datacenter :( :S
i have transfer my domain to other hosting
peace

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Shared web server ‘portvila’ moved to new hardware (RESOLVED)
3:06 am

Shared web server ‘portvila’ moved to new hardware (RESOLVED)

Posted (January 20th, 2012 at 3:06 am PST) by chalise

Due to hardware issues on shared server ‘portvila’, we have moved the server on to new hardware. The data has already been copied over, and after one final sync, sites should be back up and running again. We estimate this will take approximately one hour.

Please note, this is regarding the shared web server ‘portvila’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Update Jan 20th, 4:05am PST: All sites and services on ‘portvila’ are back up and running normally once again. Thank you for your patience! If you have any further issues, please be sure to contact support. – Chalise.

This entry was posted on Friday, January 20th, 2012 at 3:06 am and is filed under General Outages, Single Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Shared web server ‘portvila’ moved to new hardware (RESOLVED)”

My site is NOT online so it’s NOT RESOLVED. Please tell me when I can get into my host!!!!

My client’s site (licadd.com) has been running slow the last few days and FTP access has been slow as well. After several emails back and forth to the helpful support team — including running a traceroute from my computer to the website and sending the results — licadd.com has been completely down for the last half hour or so and FTP access is totally offline. Hopefully this means the DH elves are fixing the issue. Please…

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Intermittent Mail Instability on homiemail-sub5 (RESOLVED)
11:21 am

Intermittent Mail Instability on homiemail-sub5 (RESOLVED)

Posted (January 19th, 2012 at 11:21 am PST) by brians

You are likely experiencing issues while attempting to access your email, send outgoing messages, etc. Our admins are diligently working towards diagnosing and repairing this problem. We will provide an update once more information is made available. We sincerely apologize for the inconvenience this has caused. Thanks again for bearing with us. Your continued patience is greatly appreciated.

To check if you’re affected by this, go to your DreamHost panel and load up your Account Status in the upper right corner. ‘Your Email Server’ should read as homiemail-sub5. No other mail servers or services, such as web or mysql are affected by this problem.

UPDATE Jan 26th 1:50pm PST: After continuous monitoring over the last few days, our administrators have reported no further problems or timeouts. If you run into any issues with your email, please do contact support and let us know. -Oscar

UPDATE Jan 19th 4:54pm PST: After continuous monitoring over the last few hours, our administrators have reported no further problems or timeouts. We’ll go ahead and keep this post open and unresolved just to be safe while the monitoring continues, but consider this problem unofficially resolved. If you run into any issues with your email, please do contact support and let us know. We appreciate your patience! -Seohee

UPDATE Jan 19th 3:23pm PST: Our admin team is still investigating this issue. They’re currently adding more monitoring to it, but haven’t seen timeouts as of yet. Please continue to contact our support team if you are still having issues sending or receiving. At this time mail should be working for you, but if it isn’t please let us know. -Seohee

UPDATE Jan 19th 2:30pm PST: Our admin team is still investigating this issue. Please continue to contact our support team if you are still having issues sending or receiving. At this time mail should be working for you, but if it isn’t please let us know. -Oscar

UPDATE Jan 19th 2:05pm PST:We are still looking into this issue and our trying to find the root cause of this.  are still actively monitoring this situation. At this time mail on homiemail-sub5 seems to be working, but we need you to keep letting us know if it goes down for you again. There still isn’t any conclusive information regarding the cause of these stability issues. Rest assured that we are diligently working towards finding the source and solution. Sorry for the troubles this has caused. Please check back for updates. -Oscar

UPDATE Jan 19th 1:32pm PST: We are still actively monitoring this situation. Thus far the mail service is operating at optimal performance. Please understand that I am not stating that this issue is resolved. There still isn’t any conclusive information regarding the cause of these stability issues. Rest assured that we are diligently working towards finding the source and solution. Sorry for the troubles this has caused. Please check back for updates. – Brian S

UPDATE Jan 19th 12:50pm PST: This appears to be a continuation of the mail issues that we have been experiencing as of late. We are tirelessly working towards resolving this matter. Our admins have put external monitoring systems in place, which have reported sporadic outages. However, we have not been able to replicate.

We will continue to troubleshoot until we arrive at a definitive cause and solution. Sorry for all the troubles this has caused. We will update this post as information is made available. Thanks again for your patience and understanding during this time. – Brian S

This entry was posted on Thursday, January 19th, 2012 at 11:21 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

28 Responses to “Intermittent Mail Instability on homiemail-sub5 (RESOLVED)”

36 hours is intermittent?

…..“go to your DreamHost panel and load up your Account Status”……. how can I go to my Dreamhost panel if the server is down??????????????????????????????????????????????????????????????????????????????????

@Tommy: Could you please submit a support request with the details?

@Ricardo: We are referring to https://panel.dreamhost.com. That is what we mean when we say panel.

@Brians

I sent a support request about 6 hours ago.

@brians: We can’t access https://panel.dreamhost.com. Seems all dreamhost services are down!

This is getting REALLY old…our email hasn’t been right for weeks.

Also why is your chat down seemingly during the time that the most questions would be asked? That’s one thing I’ve generally been pleased about, but this is the first time chats been unavailable for me.

@brians The emails AND the web pages aren’t working. And https://panel.dreamhost.com is down as well. Please resolve it ASAP. Thank you.

My Dedicated Server, Mails and Dreamhost Panel (panel.dreamhost.com) are DOWN!!!

I’m losing money right now..

me parece increíble que todos los días tengas caídas.

@ Those of you who are reporting that you cannot access the panel.

Could you perform a traceroute to panel.dreamhost.com. I was able to successfully access through a proxy. A remote employee is not experiencing any issues with panel access either.

cannot login

panel not working for login.

what phone number can i call for support?

I’m trying to renew a domain registration. It’s been over an hour and it hasn’t been renewed. I sent a support request. I received an email confirming payment receipt, but my domain has not been renewed. And I went to this page and see that people can’t log in to the dreamhost panel so I don’t know if this is why my domain isn’t being renewed and now I’m not sure that you got my support request. Is there a number I can call?

@brians

traceroute panel.dreamhost.com

25 * * *
26 * * *
27 * * *
28 * panel.dreamhost.com (208.97.187.201) 277.842 ms *

@ Pablo

The trace looks good. You can reach our panel. What type of error are you receiving when loading https://panel.dreamhost.com into your browser?

@pablo, we need the whole trace please, including your starting IP address. You can get that by going to whatsmyip.org.

@ brians, andrea

Sorry, seems like there is also a problem with my ISP, some websites don’t load o go extremly slow..

I mean, other sites not hosted on Dreamhost..

i am getting an error msg: this session has timed out. Must reauthenticate.

Someone at dreamhost was tested my email and said it was working…. but all i hear is fart sounds….

when is going to stop all this mail failure?????? 2 weeks with problmes, change me to another server.

Mario, I just tested one of your accounts in Thunderbird and was able to send and receive without any trouble. As the recent update states, your email should be working at this time. My apologies again for the trouble. If you are experiencing issues at this time, please submit a ticket to support and let us know.

problems are intermittent, one hour works the other don’t.

Our clients are on sub4 (sub-FOUR), NOT sub5 (sub-five).homiemail, and have been having serious mail and web problems for days now.

We could always telnet into the servers, but users were getting a variety of errors including socket errors, warnings of outgoing mail held for four hours, etc.

Outgoing mail not being sent from web server?

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Hambone MySQL Downtime Due to RAID Issue (RESOLVED)
10:28 am

Hambone MySQL Downtime Due to RAID Issue (RESOLVED)

Posted (January 19th, 2012 at 10:28 am PST) by Justin@DH

The hambone mysql server is having a RAID card issue, and is not booting up. It is in our LAX datacenter and we have an admin already on his way. This is causing downtime for mysql services for customers using this shared server, which you can find in your control panel by clicking the “Account Status” link on the top right, and looking in the “Your MySQL Server” field. When we have further info, an ETA or a diagnosis we will update this post.

Update Jan 19, 1:00pm PST:The server is back up, but we are planning to move all services off of it to other mysql servers nearby. The hardware is going to be retired. The moves will not be noticable, as the data is copied while the mysql service is running. If you notice further issues please contact support. – Justin

Update Jan 19, 11:30am PST:The admin is still working on the server, but it so far has not accepted the replacement raid card either. Another one is being installed and we are going from there. When we have more info we will update this post again. Sorry for the downtime related to this issue. – Justin

This entry was posted on Thursday, January 19th, 2012 at 10:28 am and is filed under General Outages. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

7 Responses to “Hambone MySQL Downtime Due to RAID Issue (RESOLVED)”

Wondering why there could be a single point of failure like this? Can we get this setup for a RAID failure to not create huge outage? Should we not consider shared infrastructure any more redundant?

Yup, all my wordpress and other DB client sites are down. God speed, admin.

my 3rd major Dreamhost outage of the week! So glad this is what I pay for…. :/

@Ryan: The hambone mysql server itself does have a 8 drive RAID 10 system. It is one of our older mysqls though, and older technology, so we’ll be slowly retiring it in the future.

@Mizo & Kate: If your databases are on hambone, unfortunately we are still working on a fix. Sorry that you were affected by multiple, unrelated outages.

@Justin: regardless of 8 drive RAID 10 system, wondering why there isn’t a backup machine or warm backup to slide over to (using network switch). There should never be a single point of failure with share resources IMHO. That way, we wouldn’t have to wait for admin to get into data center to bring system back up. I’d like to just feel a little more secure in this since I now have some services that don’t get a lot of traffic, but people notice when its down. I heart dreamhost – just want to not have this come up again with shared servers.

If the Hambone issues is resolved, why is it that my WordPress website still can’t connect to its MySQL database?

@Chad: Your site is not on hambone, you are on spurdog. I fixed your unrelated issue.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Emergency network maintenance Thursday, Jan 19th 8pm-12am PST (RESOLVED)
11:12 pm

Emergency network maintenance Thursday, Jan 19th 8pm-12am PST (RESOLVED)

Posted (January 18th, 2012 at 11:12 pm PST) by patrickm

As a follow-up to the network outage we experienced on Wednesday and the maintenance window on Wednesday night, we will be performing more maintenance on the core routers at 8pm-12am PST on Thursday Jan 19th. At that time we will be replacing the bad line card in the core router, re-establishing the links between the two routers and then physically moving half the links back to the other core router. This maintenance should cause about 5-10 minutes of downtime.

Update Jan 19th, 11:50pm PST: Apologies for not updating this post when we began the maintenance, but it went off without a hitch! We are resolving this post now, but please feel free to contact Support with any questions! -Patrick

This entry was posted on Wednesday, January 18th, 2012 at 11:12 pm and is filed under Network Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

41 Responses to “Emergency network maintenance Thursday, Jan 19th 8pm-12am PST (RESOLVED)”

OMG!, again…and again…and again!!!

Fantastic..the morning in europe start with another one down service. 12am PST is in essence whole work day in Europe.

Ionise… this depends on what DH means with 12am. This can be either midnight or noon.

According to
http://en.wikipedia.org/wiki/12-hour_clock,
http://wwp.greenwichmeantime.com/time-zone/usa/pacific-time/convert/index.htm

8pm-12am could either be:
– 20:00-00:00 (PST) = 04:00-08:00 (GMT)
– 20:00-12:00 (PST) = 04:00-20:00 (GMT)

I guess they planned only 4 hours to do this… soo you are safe.
Maybe DH should tell us if its noon or midnight?! ;).

Does dreamhost not have redundancy built into their network for this sort of thing? As things stand I’m having to look at third party failover solutions like zoneedit and mirroring my site on another host. Easier just to move the whole site over to a provider with built in redundancy like godaddy 4GH and forget DH?

It’s more Dream than Host sometimes.

sites down for 14 hours
this is terrible for me

January 19th, 2012 at 4:04 amMarcelo Minholi Says:

Hey, it looks like DH was acquired by this guys: http://www.nouptime.com

5 hours and still no status update, and of course problem not resolved

Still no dice. Horrible! Site down for 36 hours.

NO POP SERVER AVAILABLE…… I CAN’T RECEIVE EMAILS….

Online again, tnx!

sites down 20 hours

No mail service along the day
AGAIN!!!!

January 19th, 2012 at 6:32 amNetrapal Singh Says:

Dream Host Again & Again Down ….. Today We do not have work without Mails.

@Ionise: Europe does all of its business in one minute, that just happens to be at 12:00a PST? No wonder there’s suffering.

@everyone else: According to status, all services should be up currently. If you’re having problems, maybe you should actually send in a ticket, so DreamHost knows about it.

@Matt
We are without mail service from late night until now 16:50 CET
Whole morning out of work with mails. You think this is not enough?
I remember to everyone else last days a lot users without email services for more than a week. We too.

I’m now experiencing database problems, and as such, my site is unusable. I thought there would only be downtime during 8.00PM-12.00AM PST? I’m a little disappointed, but to be honest I don’t get downtime often at all and I’m very happy with DreamHost’s customer service. Some of you should quit whining.

email sending down again (pop3), this is sporatic, will work for a while then go down, then work again. right now it’s down.

I CAN’T SEND OUT GOING E MAIL

Definitely not fixed, all my IMAP servers are offline again

This is fucking bullshit

we’re experiencing imap outgoing problems.

How can we get a status system to report or validate outages like this morning’s? (Pacific Time)

All of our hosted sites were unavailable earlier today which was brought to my attention by clients. They appear to be available now.

Hmm…. my email is down right now. Argh.

There have been almost constant email problems for the past two weeks. Can’t receive emails today. WTF is going on there?!?!?! Been with Dreamhost for over 5 years and have never experienced such ongoing crap service as this.

Still email services down
no working

homiemail-sub5 again???????

Please provide an update. I cannot receive emails yet again and working off of a blackberry is not very productive or efficient. Thank you.

Dropped out yesterday while the work was going on which was a bit of a pain, but hey.. if things go wrong sometimes it can’t be avoided. I’m still getting the connection dropping out though.

Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Request timed out.
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=161ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Request timed out.
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=161ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=159ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=162ms TTL=53
Request timed out.
Request timed out.
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=161ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=161ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Request timed out.
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=161ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53
Reply from 75.119.211.130: bytes=32 time=160ms TTL=53

an hour later, still no IMAP – looks like it may be an SSL issue, though? I can telent to mail.myserver.com 143 just fine…

My mail looks like it’s back up now…

this is fucking bullshit.

UPDATE: in my case, disabling SSL IMAP gives me access to my email. So something’s wrong with port 993 / SSL imap at the moment.

pop3 start to work for me
thank you boys. I touch wood for this will be stable.

Okay, one of my databases has disappeared but the rest are there. Is this a result of their server maintenance, or should I be worried?

Still Down, we need to wait until when?

99.99 % uptime in my as!

I have had nothing but problems since you have started the AjaXplorer……… I have no access to my website at all! Help!

Something is weird here. This thread obviously exists, but if I reload the main page http://www.dreamhoststatus.com/ this thread does not show in the main list on the left (but it does show in the “recent posts” list on the right). Confusing. Is there some problem with http://www.dreamhoststatus.com/ itself? Anyone else see this? It’s like there is some caching problem.

 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status » Blog Archive » Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups (RESOLVED)
9:25 pm

Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 18th, 2012 at 9:25 pm PST) by Seohee

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver38′, we have moved the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver38’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 23rd, 4:28pm PST: The RAID card was replaced and the restore is complete! We appreciate your patience while we got this worked out. If you are experiencing any problems at this time, please contact support and let us know. -Seohee

Update Jan 23rd, 3:04pm PST: We are replacing the RAID controller on the new hardware, due to issues which are stalling this restore. This means that this VPS host server is offline at this time. We will update this post again shortly! -Patrick

Update Jan 23rd, 1:29pm PST: Apologies for the lack of update to this post! This restore is nearing completion now! We are performing checks on the new hardware now and hope to resolve this post shortly! -Patrick

Update Jan 22nd, 9:20pm PST: The restore ran into some brief trouble as the machine was rejecting I/O. This is fixed up and the restore is continuing smoothly. We will continue to post updates until everything has been resolved. We appreciate your continued patience! -Seohee

Update Jan 22nd, 6:05pm PST: The restore is still running and our admins are monitoring the server to make sure there are no issues with the restore. We will continue to post updates until everything has been resolved. -dan w

Update Jan 22nd, 2:20pm PST: The percentage restored hasn’t increased much in the last update, but we have confirmed that the restore process is still running smoothly. It’s likely that it’s caught on a large group of small files. Our admin team is currently looking at it to make sure that it’s running as quickly as possible though. If you’re seeing any additional issues, please contact the support team. -dan w

Update Jan 22nd, 10:15am PST: Just a quick update. The restore process is now 39% done on this server. We’ll update it again in a few hours with a new percentage, or if anything noteworthy happens in the meantime. -dan w

Update Jan 22nd, 9:20am PST: Nothing new to report on this issue. The restore process continues to run smoothly and all guests are running. If you notice any problems with your services on this machine, please contact support to let us know. -dan w

Update Jan 22nd, 5:40am PST: The restore process continues to execute smoothly and we are keeping an eye on services running on this host. Further updates will continue to be posted until this issue is resolved. -JJ G

Update Jan 22nd, 2:40am PST: The restore process continues to execute properly and most VPS services have been restored at this point. Further updates will be posted as we continue to work toward the resolution of this issue. -JJ G

Update Jan 21tst, 9:58pm PST: The restore is still being processed by our admins. We will continue to post updates until everything has been resolved. -Dan J

Update Jan 21tst, 3:25pm PST: The restore is still running and our admins are monitoring the server to make sure there are no issues with the restore. We will continue to post updates until everything has been resolved. -Oscar

Update Jan 21tst, 5:19am PST: Our admins continue to manually restore the VPS services that are still not responding. This process will continue until all services are restored. We will continue to post updates until everything has been resolved. -JJ G

Update Jan 21tst, 12:40am PST: The restore process for this host machine continues, albeit slowly. We still have VPS machines that have not come back online yet and our admins are actively working on getting these restored and brought back up. Further updates will continue to be posted. -JJ G

Update Jan 20th, 6:02pm PST: Our admin team is still working to fix a few VPS machines that didn’t come back online after halting the restore process. Our admin team is processing the restores manually at the moment. We will continue to update this post as the restore progresses. -Dan J

Update Jan 20th, 2:20pm PST: We had to temporarily stop the restore process on the new machine as it was non-responsive. Our admin team is still working to fix a few VPS machines that are not working properly. The restore process has now been restarted on the new host machine. We will continue to update this post as the restore progresses. -Oscar

Update Jan 20th, 11:40am PST: The restoration process is ongoing, and services should be up and running. If you’re seeing any additional issues, please contact the support team. -Brandon

Update Jan 20th, 4:24am PST: This server’s restore is moving along nicely and all VPS guests are online at this time! -Patrick

Update Jan 19th, 11:13pm PST: Our Admin team is still hard at work on a couple of the VPS guests for this host server. We expect to have them all online shortly! Otherwise, the restore is moving along and most VPS guests are live. Please check back here for more updates! – Patrick

Update Jan 19th, 7:08pm PST: The VPS machines that are still not mounting correctly to the new machine are still being worked on by our admin team. They were able to get some up but not all. All other VPS guest machines are up and running from backups. The restore is going without any issues at this time. Please check back for updates. – Seohee

Update Jan 19th, 2:30pm PST: We have found a few VPS machines that are still not mounting correctly to the new machine and our admin team is working on getting those up and running. All other VPS guest machines are up and running from backups. The restore is going without any issues at this time. -Oscar

Update Jan 19th, 5:29am PST: This restore is well underway! We have resolved a number of issues with individual VPSs not running and things are working well now! Check back for more updates throughout the day! -Patrick

Update Jan 19th, 1:27am PST: The new hardware is now live and the backup server is mounted over NFS at this time.  With very few exceptions, all VPS guests are online at this time. We are monitoring the restore and will provide more updates throughout the night! -Patrick

Update Jan 18th, 10:22pm PST: The machine we failed onto the VPS host is having a networking problem at this time. Our administrators are looking into it now. We will continue to provide you with updates so please check this post again soon. -Seohee

Update Jan 18th, 9:41pm PST: The VPS host is now on new hardware and our administrators are building the guests and setting up mounts to serve them off of backups. We will continue to provide you with updates so please check this post again soon. -Seohee

This entry was posted on Wednesday, January 18th, 2012 at 9:25 pm and is filed under General Outages, Multiple Server Issue. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

14 Responses to “Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups (RESOLVED)”

VPS guests on ‘homie-vserver38′: ps12971 ps13164 ps15016 ps15294 ps20299 ps22224 ps22800 ps26676 ps35756 ps37854 ps38244 ps38247 ps38357 ps38677 ps43869 ps43873 ps56267 ps7789 ps7792 ps7794 ps7796 ps7798 ps7800 ps7801 ps7802 ps7806 ps7809 ps7810 ps7812 ps7813 ps7815 ps7816 ps7817 ps7818 ps7819 ps7820 ps7873 ps7886 ps8078 ps8079 ps8081 ps8083 ps8372

This is really bad… my website has the big ugly “Error establishing a database connection” webpage on it instead of my WordPress blog.

Can’t you establish virtual redundancy on this stuff? Just standup a hot server, migrate the data there/add the mount point on that server and change the DNS pointer?

@Photographer/Web Developer: Your account is not hosted on the server mentioned in this post. If this happens again, please contact Support through your Web Panel and we will be happy to investigate!

29 hours and we still do not have a usable website. Unbelievable.

We have not heard anything re: our numerous emails. our sight is STILL down. We’re losing money and patience. Please get in touch asap.

Our site is back up but we’re missing 8 days of files. :(

my site are down, and my server not it’s in the list (ps53897) answer my mail plis, open tickets, chat live, whatever pliiissss

hi dreamhost, after the move, we have a major problem with our ftp accounts, all users can access all users home directories. Please get asap in touch with me to solve the issue !

My server is STILL NOT UP and is not in this list. PLEASE CONTACT ME IMMEDIATELY. We have a number of business wanting to take legal action if this isn’t resolved today. I would like a phone call with a status update on PS43532. Thank you.

Our server went back down and its been impossible to get in touch with you, our server is ps7820.

Hi my sites are still not up. Any idea when they will be?

The communication from Dreamhost about this issue has been a joke….. it has been 3 days since the outage and i never received an email or any form of heads